DepthGraft: reusing memory made an AI model score higher

A language model is a tall stack of layers

Your text goes in at the bottom. Each layer adds a little understanding and hands the result up. The 16B model in this story has 28 layers.

What does each layer keep as it works?

Source: Table 8, p18 (28 layers).

Each layer keeps notes on every word it has read

Think of a library card catalog. Each card has a label (the key) and points to what you will find (the value). The model's version is called the KV cache, and every layer keeps its own.

What if a layer could borrow another layer's notes?

Source: Eq 1 and 2, p4. Photo credit in caption.

Sharing notes saves memory. It used to cost accuracy.

Notes take real computer memory, so some designs let neighbouring layers share one set. On the 16B model that scored 38.05 on seven tests, vs 42.38 with no sharing.

That is 4.33 points lost (computed).

Can you share notes and not lose?

Source: Table 5, p11; CLA and MLKV, p3. 4.33 = 42.38 minus 38.05.

A second worry: deep stacks water down early ideas

The authors start from this: in very deep models, what the bottom layers learn gets diluted on the way up. Picture a baton passed through 28 runners. By the end, the first runner's part is hard to find.

Earlier papers they cite make the same point (p2).

What happens when a layer reads notes written far below?

Source: Introduction, p2.

Sharing also opens a road back down

When a top layer reads a lower layer's notes, lessons from the top can travel back to that lower layer while it trains. The authors say this makes labels sharper and contents richer.

Their math needs assumptions to hold (pp21 to 22).

Does it matter which layers you pair up?

Source: Sec 3.2, pp5 to 6, Fig 2; Proposition 1, pp21 to 22.

On the 16B model, all 12 tests went up

The biggest jump is MMLU-Pro: 23.86 to 30.86, or +7.00 points. Chinese knowledge (CMMLU, +6.61), Chinese reading (C3, +6.41) and school maths (GSM8K, +5.84) follow.

The smallest is MBPP at +0.20. The average rose from 43.43 to 47.04.

Luck, or does it repeat?

Source: Table 2, p9. Gains computed from the table.

It repeated at every size, from 0.5B to 16B

Across seven model setups, 70 of 84 test scores went up and 14 went down (computed). Average gains run from +0.75 points at 0.5B to +3.68 points at 9B.

Small models gain little. At 0.5B, 3 of 12 tests went down.

Is it the shared notes, or the road back down?

B = billions of adjustable numbers. MoE = a model built from many small specialist parts. In brackets: tests that went up, out of 12.

Source: Tables 1 and 2, p9. Counts computed over all 84 scores.

Block the road back down and most of the gain goes away

The authors copied notes the same way but stopped lessons from travelling back. That scored 42.86, close to the 42.38 baseline.

Open the road and it reaches 45.30. The full DepthGraft mix gets 47.56.

Do other sharing layouts do as well?

Source: Table 4, p11. Averages over seven tests, recomputed.

Where you borrow from matters

Same tests, four layouts. Share inside small groups of neighbours: 38.05. One shared source for the whole top: 43.33. Two shared sources: 45.64. DepthGraft's matched pairs: 47.56.

FusedKV was set to start sharing at the same point as DepthGraft (p18). Its own paper reports a 50% cache cut, so this is a fair fight at one third, not at its home setting.

What does it cost to run?

Source: Table 5, p11; setup p18; FusedKV: Lin et al., ICLR 2026, arXiv 2512.03870.

Cheaper to run: a third less memory, and a sooner first word

The top third no longer stores notes of its own. 9 of 28 layers share, so about 32.1% less cache (computed).

On 16B with a 128K-token prompt, the wait for the first word fell from 2655.90 ms to 1891.80 ms. Later words got 0.64% to 0.99% slower. (A token is a small piece of text.)

So what should a reader take from this?

Source: Table 3, p10; Tables 7 and 8, pp17 to 18. Percent cuts computed.

The "why" is still a theory

Training loss, the model's score on practice text, improved only about 0.5% (p24). Yet test scores moved by several points. The authors read that as better-organized memory.

Their math holds only under stated assumptions, and their closing note asks for tighter theory (p12). With local attention, 8 of 12 scores rose and 4 fell (computed).

And how solid is the evidence itself?

Source: Appendix E.3, p24 and Fig 7, p25; Prop 1, pp21 to 22; conclusion, p12; Table 10, p23.

One number per test, private data, no outside check yet

Each score appears once. We found no repeat runs or error bars. The training data is private, and the token counts are blank in the paper's tables (pp17 to 18).

The paper is two days old and the PDF names no journal. A web search on 4 Nov 2026 found no outside replication.

Who made it, and where can you check?

Source: Tables 7 and 8, pp17 to 18; p18; vizmaya web search, 4 Nov 2026.