Close-up photo of the silicon die of an older NVIDIA GPU chip
Photo: Nvidia G71 die shot, Wikimedia Commons. An older NVIDIA GPU, not the chip tested in the paper. Author and licence on the file page.

vizmaya desk · 2 Nov 2026

Evolving Agent Teams: how AI coders get unstuck

28%That is where a strong AI coder stopped. It had days of work and a $1,000 budget. The human expert's version was 3.6 times faster.

Why did it stop there?

Source: Cao et al., Evolving Agent Teams, arXiv 2610.36848, p2 and Fig 6 (p9). The $1K budget appears on p2 only. 3.6 is computed: 1 / 0.28 = 3.57.

A kernel tells a chip how to do one job fast.

AI models run on GPUs, chips built to do many sums at once. The program that drives a GPU is called a kernel. Experts write the fastest ones by hand.

The test job comes from DeepSeek-V3, a large AI model. Score 1.0 means the expert's speed.

1.0 same as expert0.5 half as fast1.2 20% faster
A 12-inch silicon wafer
Photo: 12-inch silicon wafer, Wikimedia Commons. Chips are cut from wafers like this. Author and licence on the file page.

Can an AI coder match the expert?

Source: paper p2 (FlashInfer and MLA decode in DeepSeek-V3), p6 (speedup metric).

The best lone AI coder reached 0.6× expert speed.

Four systems tried to write this kernel from scratch, one agent each. Their best runs scored 0.28, 0.448, 0.51 and 0.602.

Budgets and run counts differ by system.

What went wrong inside the stall?

Source: pp 2, 7, 9; Fig 6; Table 2 (best of each system's runs). MLA decode on H100.

They took the easy road and never went back.

The hard design kept failing its tests, so the AI dropped it. It polished an easier design instead.

The easy design uses the chip's math hardware less fully. Its ceiling is lower.

Would a bigger team fix it?

Source: pp 2, 7 (hard design WGMMA, easy design WMMA). The chart is a sketch of the idea, not data.

A fixed team of four did not fix it.

The team had four set jobs: profiler, researcher, planner and builder. Nobody could change the setup mid-run.

Contested The paper gives two scores for this team, and it used a newer model than the team it is compared with.

55%Page 2, introduction
0.31×Page 9, Fig 6

What if the team could change shape?

Source: pp 2, 6, 7, 9, 14. Model: Opus 4.7 on pp 6 and 14; the compared team ran Opus 4.6 on p7.

The fix: a boss that rebuilds the team when it stalls.

The boss is another AI agent. When progress stops, it finds the cause and designs the next team.

Redesign means three things: who does what, how they work together, and what they must read.

What does one rebuild look like?

Source: pp 4, 5, Fig 2 (supervisor loop and shared file system).

Each rebuild answered one named stall.

Parallel work lifted the score 44% from V5 to V6. Fresh measuring lifted it 32% from V9 to V10.

The run ended at 0.96 on this scale. That is 1.2 times the production kernel, which sits at 0.77.

What happened between V4 and V8?

Source: pp 1, 5, 6, Fig 1. +44% and +32% re-computed: 0.49/0.34, 0.66/0.50. V2 left out (see slide 12).

One good tip sat in the wrong file. Three teams missed it.

Team V4 wrote its best idea in its own report. The shared notes did not hold it, so V5, V6 and V7 never saw it.

The boss added a note-keeper. One plan now joined every finding, and every team had to read it.

Did the rebuilds add up to a win?

Source: p5 (V7 to V8 case study). "First P@V WGMMA success ever" is the paper's own wording.

Same model, same budget: the rebuilt team won all four jobs.

Best of two runs each. A score under 1.0 is slower than the expert kernel.

Two jobs went above 1.0: GDN at 1.90× and GEMM+AR at 1.18×.

Can it beat the expert outright?

Source: p7, Fig 3a. Both use Claude Opus 4.6. "Matched budget" is the paper's term (pp 1, 7).

1.21×

With more budget, it beat the production kernel.

It ran faster on 36 of 47 real workloads. On the other 11 it ran slower, at 0.86× to 0.99×.

Where does this break?

Source: pp 1, 2, 10 (1.21× mean speedup, 47 FlashInfer-Bench workloads).

Averages hide losers. On GDN, 43 of 100 jobs ran slower.

GDN averaged 1.90×. Many big wins came from small jobs, where the old kernel wastes time on setup.

With more budget, 37 of 100 stayed slower. The lone Claude Code run had 49 of 100.

Each main setting ran twice. Best run shown.

Which numbers does the paper give two ways?

Source: pp 6, 7, 8 (GDN prefill, 100 workloads; best run each).

Five numbers the paper gives two ways.

What
One place
Another place
Fixed four-role team
55% p2
0.31× p9
Model behind that team
Opus 4.7, pp 6, 14
"same model", p9
Rebuilt team, MLA job
0.74× pp 2, 7, 9
0.75× p7; 0.721×, 0.685× Table 2
GEMM+AR, extra budget
1.49× p8, section 4.2
1.56× p8, section 4.3
Best earlier system
0.61× p2
0.602×, 0.598× Table 2

We left out one more: the paper's V2 score of 0.065, which sits below V1's 0.15 on a best-so-far line.

So what should you take from it?

Source: pp 2, 6, 7, 8, 9, 14; Table 2. Preprint, not peer reviewed.

When you are stuck, change the team, not only the effort.

The paper's loop has three moves. Name the exact failure. Change who does what. Write the lesson where the next person will read it.

Our read of the paper. The authors tested this on AI agents, not on people.

Rows of server racks in a data center
Photo: Datacenter Server Racks, Wikimedia Commons. Author and licence on the file page.

Where does every number come from?

Source: pp 4, 5, 15 (supervisor loop; failure was structural, not capability, in Appendix B).

Sources and credit

The paper

Method

  • Every number comes from the paper, matched to its page. Computed: 1 / 0.28 = 3.57; 0.49 / 0.34 = 1.44; 0.66 / 0.50 = 1.32; 0.96 / 0.77 = 1.25.
  • Not checked: model names, and outside replication. One search found none.

Photos, Wikimedia Commons

Credit

Research by the paper's authors. Compute support listed on p11: SemiAnalysis, Amazon, AMD and NVIDIA. Story by vizmaya desk.

Pages cited are the printed pages of the PDF (1 to 17).