Vizmaya Labs · Paper explainer · 4 Nov 2026

Teach an AI one new trick. Watch it forget the others.

A new paper says it knows why this happens. It also offers a fix. Then its own table shows the fix removes about a quarter of the damage.

Scroll. By slide 7 you will see exactly where the damage goes.

1 · The problem

It happens to nearly every model that gets a job.

A big model first learns from a huge pile of text. Then a company trains it again on one small job, like answering support tickets or solving math.

After that second round, the model is often worse at things it used to do well. The paper calls this Pervasive Capability Degradation. We will say forgetting.

Illustration only, not paper data

Math

Grammar

Trivia

Top bar: before the second round. Bottom bar: after.

But a model has no shelf where old facts sit. So what exactly gets erased?

2 · Where ideas live

The model's mind is a packed suitcase.

It knows far more ideas than it has room for. To make them fit, it stores each idea as a direction, like a spoke on a wheel.

Ideas that rarely show up together can share space. So the spokes sit almost at right angles, and barely disturb each other.

The paper borrows a name for this packing: superposition.

Overstuffed suitcases — Photo: Wikimedia Commons, Suitcases-1.jpg

3 · What practice does

Practise one thing, and ideas get glued together.

In the original training, ideas show up one at a time. In a narrow job, the same few ideas show up together, every time.

The paper's theory says the model then merges them into one spoke. That saves room. Watch it happen.

Theory, borrowed from toy models. Not measured directly in this paper.

4 · The leak

A glued idea leaks.

Now there are fewer spokes, and they are no longer at right angles. Ask about idea A, and idea B lights up a little too.

The paper calls this a ghost signal. It is faint, but it keeps travelling through the layers that follow.

5 · The last step

Every possible next word owns a territory.

At the end, the model turns its inner state into one next word. Picture that state as a dot on a map. Each word owns a patch of the map.

Whichever patch holds the dot is the word the model says. A leak nudges the dot. Nudge it across a border and the answer changes.

Simplified map. The paper uses a weighted version of the same idea, called a Laguerre-Voronoi diagram.

the

Or drag the dot yourself.

Coloured Voronoi diagram — The map idea, drawn flat. Wikimedia Commons, Coloured Voronoi 2D.svg

John Snow cholera map 1854 — Same habit of mind, 1854: John Snow's map of which water pump sits nearest each death. Public domain, Wikimedia Commons.

Reticulated giraffe — Patches split by borders, in nature. Wikimedia Commons, Girafe réticulée 1.jpg

6 · The evidence

Train on yes and no. Get yes and no everywhere.

The authors trained a model (Mistral-7B) on a task whose answers are the words yes and no. Then they tested it on ordinary text from the original training mix.

With plain training, the end-of-text mark newly lost to "yes" or "no" in 10,457 comparisons.

Plain training: newly flipped comparisons, end-of-text vs

" yes"

4,327

" no"

3,474

"yes"

1,927

"no"

729

With the paper's fix (AEGIS)

yes or no

none in top 10

Source: Table I1 and text, arXiv 2611.05003. Bar length is scaled to the largest bar.

7 · The one number

A quarter to a third.

0% to 0%

That is how many newly flipped comparisons the fix removes, compared with plain training. It was measured on one 7B model (Qwen2.5-7B) after training on three different jobs.

Millions of newly flipped comparisons

RTE, plain

5.700

RTE, fix

4.309

GSM8K, plain

2.321

GSM8K, fix

1.546

CodeAlpaca, plain

1.941

CodeAlpaca, fix

1.370

Source: Table 5, arXiv 2611.05003. Cuts vs plain training: 24.4%, 33.4%, 29.4%. The paper's own headline compares against the average of six methods: 25.7%, 35.4%, 29.9%. I recomputed all of them.

8 · The catch

Most of the damage is still there.

On the RTE job, 5.700 million new flips drop to 4.309 million. That leaves about 76 in every 100.

The abstract talks about "fully resolving" the problem. Table 5 does not show that.

Context: the base model, before any training, already has 18.569 million flipped comparisons. Packing ideas tightly always leaks a bit.

24 squares removed by the fix. 76 remain. Each square stands for 1 in 100 of plain training's new flips on RTE.

9 · The price

About 1.3 times the compute. With one catch.

Memory barely moves: 3,458 GiB against 3,430 GiB on the 72B model. Compute holds steady at about 1.27 times.

Time is less tidy. The smallest model needs 21.1 GPU hours against 10.2, which is 2.07 times.

Training time, fix vs plain (1.00 = same)

Qwen 3B

2.07x

Qwen 7B

1.26x

Qwen 32B

1.36x

Qwen 72B

1.31x

Compute (FLOPs), fix vs plain

All four sizes

1.26x to 1.27x

Source: Table 4, arXiv 2611.05003. The "plain" run here already includes the extra anchor data, so the true add-on cost is measured against a baseline that has paid for it. The fix also needs a one-time pass over the base model first.

10 · Before you believe it

Six reasons to wait for a second lab.

It is one day old. Nobody has rerun it. The PDF itself says "Under review".
The yardstick is already strong. Plain training gets the extra anchor data too, which shrinks the gap. That is the fair choice, and the one we used.
Two of the six rivals are weak. They are model-merging tools that plain training beats in most columns.
One rival is the authors' own earlier method. The new one wins by small margins in many cells.
Run-to-run noise is hidden. The paper says it is "below 0.03" and leaves it out. Many gains are smaller than that.
Accuracy shows up on 7B to 9B models only. The 3B to 72B runs measure cost, not quality.

3 Nov 2026Posted as a preprint

NextSomeone else reruns Table 5

ThenCode check, bigger models, peer review

11 · What to take away

The leak is real. The fix is partial.

If you tune a model for one job, test it on jobs you did not train for. Then look at which words it now says too often. After a yes and no job, this model reached for yes and no everywhere.

Whether this particular fix is the best one will take other labs a few months to settle. The leak itself you can check this afternoon.

Verdict: a real leak, about a quarter of it plugged.

Paper: arXiv:2611.05003, Unifying and Beyond the Theory of Pervasive Capability Degradation in LLM Post-training SFT from Feature Interference (Chen, Zhu et al.), arXiv preprint, 2026-11-03, https://arxiv.org/abs/2611.05003. Picked from AI Daily's papers, featured in Doom v Boom on 2026-11-04 (https://vizmaya.fyi/ai-daily/doom-v-boom/2026-11-04).
Numbers come from the paper's Tables 4, 5 and I1 and its text. Percent cuts against plain training are our arithmetic on Table 5. The wheel and map visuals are simplified illustrations of the paper's ideas, not its data.
Photos: Wikimedia Commons, files Suitcases-1.jpg, Coloured Voronoi 2D.svg, Snow-cholera-map.jpg, Girafe réticulée 1.jpg. Check each file's page for the exact author and licence before publishing.