The AI that keeps typing after it has already asked for help

The old world

An AI agent works in turns.

It thinks. It asks a tool for help, like a search engine. Then it has to wait for the answer.

Does it stop writing the moment it asks?

Source: p1. Illustration, original.

First principles

This kind of AI writes in blocks, not word by word.

It starts with a block of blank slots. Each round, it fills in many slots at once.

What if the tool call ends halfway through a block?

Source: p1, p3. Illustration, original: 16 slots, 4 rounds.

The crack

The block does not know when the question ends.

The model keeps working on the whole block. Everything after the tool call gets computed, then thrown away.

Contested detail

The paper says 40 positions were discarded in its example. Its figure implies 35 (48 minus 13, computed). It does not explain the gap.

Could the model learn where to stop?

Source: p1, p8, Fig 4. Illustration, original.

The turn, part one

Trick one: shrink the window.

A small controller marks where the useful part ends. Work past the marker stops. If the answer needs more room, the window can grow back.

The paper calls this flux decoding. In its example, 48 slots became 13.

And what about the easy words?

Source: p5 (Eq. 4), Fig 4 p8. 72.9% is computed from 48 and 13.

The turn, part two

Trick two: fill in the easy parts all at once.

Tool calls use a fixed format. The controller spots those pieces and writes them in one step. Hard parts still get normal care.

The paper calls this confidence alignment decoding.

Who teaches the controller?

Source: p4 (Eq. 3). Illustration, original.

The turn, part three

A small coach, trained with a simple reward.

Wrong answer: no reward. Right answer: more reward the less work it took.

The big model stays frozen. The coach is a two-layer network, trained on 8 NVIDIA A100 chips for 40 rounds per stage.

Did the coach beat the other ways to go faster?

Source: p4 to p6, Eq. 5. Bar lengths show the idea, not the paper's numbers.

The result

Same model. 24.6% to 43.1% less work.

On all five tests, the model with the coach used less compute than the same model without it.

Is that good against the cheapest rival too?

Source: Table 1 p6, units x10^12 operations. Reductions computed from the table.

The result, tougher test

Against the cheapest rival, it still cut 20.6% on average.

On each test, the authors compared against whichever rival used the least compute. The smallest win was 7.2%. The biggest was 43.1%.

Which trick does the heavy lifting?

Source: p1, Table 1 p6. Percentages computed from the table.

Taking it apart

Stopping early matters most.

Switch off the shrinking window and the work goes up 24.5% on average. Switch off the easy-fill trick and it goes up 16.3%.

Did the answers get worse?

Source: Table 2 p7. Averages over five tests, computed.

The catch check

Accuracy held. Mostly.

On the main model, accuracy moved from minus 0.4 to plus 4.4 points. On a second model, from minus 2.2 to plus 1.72.

So where does it break?

Source: Table 3 p8. Points are percentage points. Dream cut 17.2% to 67.5% of the work.