A fork in the road, where one path splits into two

vizmaya desk · 3 Nov 2026

SpecOPD: how AI guessers learn from their own mistakes

A 2026 paper teaches a small AI to guess ahead. It trains on its own guesses, and a big AI grades them all at once.

Why would a better practice score make a worse guesser?

Photo: "Fork in the road, decision tree, September" by Wonderlane / Flickr, CC BY 2.0. Paper: arXiv:2611.02846v1.

The surprise

A better practice score. A worse guesser.

4.6177words kept per check, before training
4.4708after training on the big model's answers

The practice score kept improving. The guesser's real score fell.

Trained a new way, the same guesser rose to 4.7822.

First, what is a guesser, and why does anyone need one?

Source: paper p8 (Fig 8), p17 (Table 2). Test setup: mid-size target, DFlash V2 drafter, sampling at temperature 1. Training loss fell in all three runs.

Big AI models write one word at a time.

Ask a chatbot a question. It writes the answer in small pieces, called tokens.

Each piece waits for the one before it. That wait is the slow part.

Could a small helper write ahead?

Source: paper p1, which cites the 2023 speculative decoding papers by Leviathan et al. and Chen et al. Illustration made by vizmaya, not data.

Close view of the keys of an old typewriter
Photo: "Corona Standard typewriter keys, c.1939" by mpclemens / Flickr, CC BY 2.0

A small guesser races ahead. A big model checks.

The guesser proposes the next seven pieces at once. The big model checks all seven in one go.

It keeps the guesses up to the first miss. A longer run means a faster answer.

How do you train a guesser to last longer?

Source: paper p1, p7 (seven draft tokens). The paper's score, called acceptance length, also counts one bonus token. The chips are our illustration.

The old way: practise on the answer key.

The usual training copies finished answers written by the big model.

But in real use, the guesser continues from its own guesses. The two paths drift apart.

What if it practised on its own road?

Source: paper p2 (fixed reference answers), p8 (off-policy baselines).

A road that splits into two at a fork
Photo: "Fork In The Road" by Ian Sane / Flickr, CC BY 2.0

The crack: practising on its own guesses is slow.

It works better. But every guess must be checked before the next one can start.

In the older own-guess method, recorded GPU activity was just 26.37%. One update took 12.99 seconds.

Do guesses really have to wait in line?

Source: paper p1 (Fig 1), p8, p19. Recorded activity is not full chip use, and covers four updates only. The older method is called Draft-OPD.

Rows of server racks in a data center
Photo: "Virginia Tech - data center" by cbowns / Flickr, CC BY-SA 2.0

The turn: grading a guess needs less than we thought.

The big model needs two things. The true text up to that point, and the guesser's own earlier guesses.

It does not need a whole finished answer first. So guesses can start from many places at once.

Can many guesses share one start?

Source: paper p2, p3 (Fig 2). The sentence is our illustration, not the paper's example.

Many guesses, one shared start.

The big model reads the shared start once. Then it grades every branch from it.

103.45×fewer rows to process, in a paper calculation

Did training actually get faster?

Source: paper p24 (computed: 529,664 / 5,120 = 103.45). Assumes a 2,048-piece answer and 512 start spots. A paper count, not a stopwatch.

A rope being twisted from several strands
Photo: "Turku Medieval Markets, twisting rope" by Samuli Lintula / Wikimedia Commons, CC BY 2.5

The result, part one

Training ran faster.

16.09×graded tokens per second, 28.42k vs 1.77k

Updates came 4.95× faster. Each graded 3.25× more tokens. Recorded GPU activity rose from 26.37% to 77.23%.

But did faster make better?

Each method counts its own graded tokens. The paper does not measure time to reach the same quality. Ratio computed from rounded values: 16.06×.

Source: paper p1 (Fig 1b), p8 (Fig 7), p19. Eight GPUs, mid-size target.

The result, part two

And a better guesser.

Trained on its own guesses, the guesser went from 4.6177 to 4.7822 words kept per check.

Trained on the answer key, it fell to 4.4708. All three runs lowered their training loss.

Caveat

SpecOPD used 32 GPUs. The answer-key runs used 16.

So how much faster do answers get?

Source: paper p17 (Table 2), p8 (Fig 8), p16. Mid-size target, temperature 1. The answer-key line mixes two methods: E2E-TV at 8k, CE at 10k and 12k. Axis starts at 4.0.

The payoff

Answers came up to 3.56× faster.

Best case, against writing one piece at a time. On average 2.97× with one request at a time, and 1.35× with 32 at once.

Next time a headline says "× faster", ask: faster than what?

Faster than what, exactly?

Streaks of light from moving traffic at night
Photo: "Light Trails" by wwarby / Flickr, CC BY 2.0

Source: paper p7 (Fig 6), p18. Mid-size target, one request at a time. Values read from the figure's labels; workload names read from bar order. Averages are geometric means.

Bigger models, smaller gains.

The biggest jump was on the smallest setup: +41.45%. On the mid and large targets it was +2.72% and +4.59%.

On the mid-size model's coding-agent tests the guesser barely moved: +0.04% and +0.06%.

Contested

The authors guess the largest guesser lacked agent data early on. They did not test that.

And against the best rival?

Source: paper p6 (Table 1, computed from the table), p7. Temperature 0, average of five tasks. Target sizes are as reported by the authors: 8B, 27B and 125B.

Faster than the best rival? Barely.

Against the earlier guesser, DFlash V2, output speed rose just 2.83%, 2.20% and 1.21%.

One test got slower: software fixes with 32 requests at once, down 4.46%.

What did the paper not test?

A hand-held stopwatch
Photo: "Stopwatch" by wwarby / Flickr, CC BY 2.0

Source: paper p18 (A.8). One run per condition. Mid-size target. Speedups over plain writing, as geometric means of five workloads.

The limits

What the paper did not test.

  • Time to reach the same quality. Only speed of training was measured.
  • Equal compute. SpecOPD ran on 32 GPUs, the answer-key runs on 16.
  • Repeat runs. Speed tests are one run per condition.
  • Clean test data. 64 Terminal source trajectories overlap the training data.
  • Peer review. It is a one-day-old preprint, by authors at the firm that builds the models.

So who did the work, and how can you check it?

Source: paper p16 (A.6), p14 (A.1), p18, p19 (B.2), p1. No answer-quality scores were found in the paper (our reading, a gap).

Credit

Who did the work.

Tianyu Liu, Jun Zhang, Wenjie Huang, Yuhao Shen, Tianqi Xu, Xinyue Hao, Hongyaoxing Gu, Xuyang Liu, Baolin Zhang, Yating Zhu, Shuang Ge, Jun Dai, Lei Chen, Yue Li and Mingchen Wan.

Qwen Business Unit of Alibaba, with six universities. The first three authors share equal contribution. Jun Zhang is the corresponding author and project lead.

The paper builds on speculative decoding (Leviathan et al., Chen et al., 2023), Draft-OPD, the E2E-TV loss, DFlash and tree attention.

Check anything. Every number has a page.

Source: paper p1, p9, references.

Sources and method

Sources

Photo credits