GMT+8 SG --:-- 0000 X 0000 Y /
← 全部文章← All writing

Agents

How do I know the agents actually made my team faster?

Everyone on the team uses agents every day to write code, run experiments and chase down problems. Ask "did they make us faster?" and the answer is always "yes". That answer carries no information. The real question is: faster where, slower where, and how would I know.

Code volume is the easiest number to measure, and the easiest to be fooled by

The first metric you can get is output: lines of code, commits, experiments. Those numbers will rise, because what agents do best is make the "write it down" step fast. But written is not correct, and correct is not worth doing. A team that watches only that curve ends up somewhere very busy with an increasingly blurry sense of direction.

So I stopped asking "how much" first and started asking "what".

Classify the work before you talk about speed

I borrowed a public taxonomy that splits R&D work into six phases: deciding what to work on, designing the approach, building code and data, running it, analysing results, communicating conclusions. Then I sorted every task the team handed to agents into one of the six.

Two things showed up immediately. First, agent output is overwhelmingly concentrated in building and running, especially troubleshooting infrastructure. Things people used to wait for a colleague to help with, they now ask the agent. Second, deciding and designing were almost never delegated, and shouldn't be. What accelerated was execution, not judgment.

That conclusion isn't novel, but measuring it changed how the team talks. When someone now says "let the agent do it", the first question is "which phase is this?" The last four: hand it over. The first two: decide who owns it first.

Intervention rate is the real signal

The second useful quantity is the intervention rate: how many times a person stepped in between handing a task over and getting something usable.

It's more honest than success rate. Success rate rises as tasks get easier; intervention rate doesn't. A task spanning several hours, even when it ends in success, has most likely been pulled back once or twice along the way. Those interventions are the boundary of the system's capability, and where the boundary sits matters more than how fast things run inside it.

I asked the team to log interventions, not to score anyone, but to see patterns. When the same kind of task keeps needing a human at the same point, a constraint or a tool is missing there. The right move then is not to keep stepping in, but to turn the step-in into a checker or a tool so the next run doesn't need a person. This is the same idea as "the report is not the endpoint" that I wrote about elsewhere.

Measurement itself has to be restrained

Two traps to avoid.

One is turning measurement into appraisal. The moment intervention rate touches performance reviews, people log fewer interventions and the number is worthless. It can only be a mirror the team holds up to itself, never a ruler for managing people.

The other is mistaking the easy-to-measure for the important. Output volume is the easiest to measure; quality of judgment is the hardest, and it's the latter that decides where the team ends up. A dashboard that measures only the former systematically steers resources toward whatever looks busiest.

So my dashboard now has three rows: the distribution of tasks across the six phases, the intervention rate in each, and how many of those interventions have since become checkers. The third row matters most. It records whether the team is getting stronger or repeating itself.

Conclusion

Whether agents accelerated the team can't be answered by feel, and can't be answered by code volume. Classify the work first, then count how often a person had to step in within each class, then count how many of those step-ins have been hardened into the system. After those three steps, the answer is no longer "yes". It's a map of where to invest next.