SpecPilot: how an AI loop sped up LLM decoding 2.13x

The old world

A chatbot writes one piece at a time

Text comes out in small pieces called tokens. Each new piece needs one full trip through the model.

On this chip, plain decoding makes 49.9 pieces a second. That is 20.0 ms for each piece (computed: 1000 / 49.9).

Illustration of tokens, not data. Source: p1; Table A1, p21

The crack

The chip mostly just reads

To make one piece, the chip reads all of the model's numbers from memory. That read takes 80% of a 23.61 ms step.

So extra checking is almost free. The main math took 17.88 ms for 1 row and 18.58 ms for 64.

That is the trick behind speculative decoding. A small helper guesses ahead. The big model checks every guess in one pass.

Source: p5; Fig 2a. 0.70 ms gap computed (18.58 - 17.88).

The crack

Stock checks one chain, then quits at the first miss

The helper ranks several guesses for each spot. The stock system follows just one and stops at its first miss.

A tree keeps the runner-up guesses alive. In replay on the same guesses, a chain locks in 4.25 pieces per check. A 32-node tree gets 5.38.

Source: p5, p8 (Fig 6), p13 (Fig 10). Diagram is an illustration.

Illustration, not data. Green is accepted, red rejected, dashed never checked.

The turn

So the authors built a loop, not a fix

An AI agent loop measures where speed is lost, tries fixes in parallel, and keeps only those that win on requests it has not seen.

Over 13 rounds it made 486 code changes. 54 survived: 11.1% (computed). The agents were Claude Opus 5.5 inside Claude Code.

The turn

A speed-hungry agent can cheat. So every word gets checked

A faulty change can nudge the model's choices and still pass an accuracy test. Its accuracy moved by -1.6 points, with a range of -3.6 to +0.3 that includes zero.

Token forcing re-scores each output word against the original system. Honest systems matched stock (0.73% vs 0.72%). The faulty one did not (2.0%).

Source: p13 (section 6.2); p6 (Fig 4). Limit line = stock + 0.5 points, computed.

Round 1 of 3

First, use the guesses already on the table

Three changes: a cheaper helper, a tree, a wider tree. Speed went from 184 to about 270 tokens a second.

Then the loop re-measured. A better tree on the same guesses would add little. The bottleneck had moved.

Source: p12; Fig 8; Table A3 (p21). Step values computed by adding the gains printed on Fig 8.

The twist

Add a crowd and the speed-up shrinks

Each extra request brings its own work, so reading the model stops being the main cost.

Stock's speed-up over plain decoding fell from 3.40x to 1.89x at 24 requests. On a smaller model it fell from 3.73x to 1.49x at 32.

Source: p3 (Fig 2b); p5.

What made the difference

Re-measuring beat just trying things

A loop with no cost model reached 1.34x over stock. The full loop reached 2.13x.

The plain loop kept adding tokens per step. It never touched weight reads, which took 80% of the step. The full loop re-measured every round and followed the biggest gap as it moved.

One comparison. The paper does not state how many runs.

Source: p12 (section 6.1).