Speculative decoding energy · vizmaya desk · 2 Nov 2026
When the speed trick backfires
A trick makes AI write 1.3 to 4.3 times faster for one person. Researchers in Germany measured what it does to the power bill when more people join.
Source: Ma et al., arXiv:2610.40287, preprint of 29 Oct 2026. Speed range: p.2 and Table 4.
How much faster, exactly?
The hook
tokens a second, written for one person
A token is a word or a piece of a word. One big AI model wrote about 30 of them every second. With one trick, it wrote about 128.
The trick is called speculative decoding. The paper asks what it costs in electricity.
Source: paper Table 4, p.11 (gemma-4-31B-it: 33.2 ms vs 7.8 ms per token). Computed by vizmaya desk: 1000 ÷ ms.
Why does electricity matter here?
A datacenter has a fixed power supply.
The paper starts here. The energy used per token decides how many tokens a datacenter can write for people.
Save a little on every token, and the same building serves more people.
Source: paper abstract and p.1
Where does the energy go when an AI writes?
An AI writes one token at a time. Each one needs the whole model.
The model is a huge set of numbers kept in the chip's memory. To write one token, the chip reads all of them.
Then it reads all of them again for the next token.
Source: paper p.2
Is reading that memory the costly part?
Moving data costs 186 times more than using it.
On a chip from the same family as the ones tested, reading one byte from memory costs about 93 picojoules. One 16-bit math operation costs about 0.5.
A picojoule is a tiny unit of energy. The ratio is what matters: 93 divided by 0.5 is 186.
Source: paper p.2, citing Antepara et al. (SC '25). Ratio computed by vizmaya desk.
How do you read the memory less often?
Fix one: let many people share one read.
Serve many people at once and the chip reads the model once for all of them. This is called batching.
For the model gemma-4-31B-it, energy per token fell from 12.65 joules with one person to 0.521 with 64. That is about 24 times less.
Source: paper Table 3, p.6 (no helper). Ratio computed: 12.65 ÷ 0.521.
Does sharing make anyone's answer faster?
Sharing saves energy. It does not make you faster.
On the same model, each person waited 33.2 milliseconds between tokens when alone.
With 64 people sharing, each waited 56.6 milliseconds. The energy got cheaper and the wait got longer.
Source: paper Table 4, p.11, and p.2
Can anything speed up one person?
Fix two: guess ahead, then check.
A small helper guesses the next few tokens. The big model checks all the guesses during one read of its memory.
Right guesses are nearly free. At the first wrong guess, the rest are dropped, and the model adds one token of its own.
Source: paper p.2 to p.3. The example words are made up to show the idea.
Does the trick save energy for real?
For one person, it works. The median best saving is 54%.
The authors tested 11 models and 5 kinds of helper. On every model, at least one helper saved energy for a single person.
The best saving ranged from 15% to 77%.
Source: paper Table 2, p.6 (batch of 1); median on p.5
What happens when more people join?
Add people, and the saving melts.
Qwen3-4B went from a 56% saving for one person to 19% for 64. Qwen3-14B went from 54% to 18%.
Sharing had already done the job the helper was hired for.
A few models did the opposite. Qwen3-Coder-30B-A3B went from 32% to 51%.
Source: paper Table 2, p.6
Why would a helper ever cost energy?
A wrong guess still gets checked. Checking costs.
One helper, called DFlash-16, checks 16 tokens for every person at every step. It keeps fewer than 3.
With one person the chip has room to spare. With many people, each wasted check takes room from real work.
Source: paper p.3 and p.15
How crowded is too crowded?
Past a tipping point, the saving becomes a loss.
A chip can be slowed by memory or by math. One person uses about 10 math steps per byte read. The tipping point is near 316 on an H100 chip and 163 on an H200.
Past that point the saving flips to a loss within one doubling of users.
Source: paper p.2, p.14 to p.15, Fig. 6 (260-token context)
So when should you use the trick?
Guessing buys speed that sharing cannot.
A service often promises a speed, such as 25 milliseconds between tokens. On gemma-4-31B-it, the normal setup misses it at every batch tried.
With guessing, one person gets 7.8 ms and 32 people get 19.0 ms. At 64 people it slips to 28.9 ms. For energy alone, the paper says add people first, then check that a helper still pays.
Source: paper Table 4, p.11, and p.17
Does it work on every kind of text?
Test it on the text you actually serve.
A helper trained for code kept 5.9 tokens per check on code. On chat it kept 2.0 to 3.0.
At 64 people it saved 51% of the energy on code. On chat it cost energy.
Source: paper p.8 (Qwen3-Coder-30B-A3B-Instruct with its DFlash helper)
What could still make these numbers wrong?
A result can look right and still be wrong.
- Energy was counted on the chip board only. Cooling and the main processor are left out.
- Batches stop at 64 people and the prompts are short.
- Not peer reviewed yet. ContestedAt long contexts, other work says the saving can grow with batch.
Source: paper p.5, p.15, p.16. The contested point cites MagicDec (p.5).
Who did the work, and where can you read it?
Sources & method
The paper
- Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets, arXiv:2610.40287, 29 Oct 2026, CC BY 4.0.
- More stories: vizmaya stories.
Credit
Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler and Gerhard Wellein, Erlangen National High Performance Computing Center (NHR@FAU), Germany.
Method
Every number comes from a page or table of the paper. Three are computed by vizmaya desk: 186 (93 ÷ 0.5), 24 (12.65 ÷ 0.521) and 30 vs 128 tokens per second (1000 ÷ ms).
The paper is a preprint. Nobody has replicated it yet.
Images
All drawings are original, by vizmaya desk. No photographs are used.
vizmaya desk · 2 November 2026