vizmaya desk · 2 November 2026
How AI search learns from its own wrong turns
By step 3 of a four-step question, 70% of the questions the tool is still working on already carry one of its own wrong passages.
How does a search tool end up there?
Source: Qiu et al., arXiv 2610.40409, Sec 3.4, Fig 3a
How it works
Hard questions need a trail of passages.
A passage is a short chunk of text. A four-step question needs four of them, found in order.
At each step the tool reads the question plus its notes so far. Then it picks the next passage.
How do you teach a tool to follow a trail?
Source: Qiu et al., Fig 1, Sec 3.1
The old way
Practise with perfect notes.
In training, the tool's notes hold only right passages. It just learns to pick the next right one.
This is called teacher forcing. It works well, and it is how chain search tools are usually trained.
Every note in practice is clean.
But what does it hold when it works alone?
Source: Qiu et al., Sec 3.3, Fig 1a
The crack
Live, the notes hold its own picks.
Nobody hands it the right passages. Once it picks a wrong one, that passage stays in its notes for every later step.
The study calls each set of notes a state.
How often does that happen?
Source: Qiu et al., Sec 1, Sec 3.3
How often
By step 3, most notes already hold a wrong passage.
Four-step questions, tool trained on perfect notes: 19% of notes at step 2, 70% at step 3, 89% at step 4.
On three-step questions it is 12% at step 2 and 48% at step 3.
Does a wrong passage really cost that much?
Source: Qiu et al., Sec 3.4, Fig 3a. 1,000 MuSiQue test questions; notes still missing a right passage.
The cost
Clean notes: 67% success. With a wrong passage: 28%.
Success means finding a right passage that was still missing, on the next step.
The gap shows up on two-step, three-step and four-step questions.
Maybe those notes were just harder. Is there a fair test?
Source: Qiu et al., Sec 3.4
The fair test
Delete the wrong passage and success jumps 11.3 points.
The study took 1,358 notes that held a wrong passage and deleted it. Nothing else changed.
Success went from 20.8% to 32.1%. The mistake alone costs about a third of the tool's remaining chance.
What if the tool practised on its own mistakes?
Source: Qiu et al., Sec 5.4.2. Check: 32.1 minus 20.8 = 11.3.
The turn
The fix: practise on the trail you really walk.
The answer key still knows which right passages are missing, even when the notes are messy. So every set of notes gets a clear label.
Nothing else changes: same tool, same scoring rule, same index, same number of examples.
Does it work?
Source: Qiu et al., Sec 4.1 to 4.3, Fig 1b, Algorithm 1
The result
After two or more mistakes, it recovers twice as often.
Both tools were scored on the same 3,403 notes. With a clean start they tie: 67.0% and 69.0%.
After two or more mistakes it is 29.2% against 14.5%.
Is that just fake mistakes thrown into practice?
Source: Qiu et al., Sec 5.4.1, Fig 3b. Shared seed.
What matters
Fake mistakes give about half the gain.
Adding made-up wrong passages lifts full answers from 44.6 to 47.9. The rest of the gain needs the tool's own notes, refreshed every round.
Reusing one old walk gets 48.4. Refreshing it gets 50.5.
How does that stack up against other tools?
Source: Qiu et al., Table 4, Sec 5.5.1. One training run each.
The scoreboard
It fully answers 49.3% of hard questions. The best rival: 36.5%.
Fully answered means every needed passage is in the top five.
Search-R1 calls a 7B language model about four times per question. The retrained tool calls none.
Does it hold on longer chains it never practised?
Source: Qiu et al., Table 1 (five-seed mean), Table 9. 1,000 MuSiQue questions.
Longer chains
On trails it never practised, it still held up.
It trained on 1,800 questions only. Tests used three to five steps over text collections 19 to 48 times larger.
Plain search fully answered 1.0% and 2.0%. Search-R1 wins the second test, 37.0 to 32.8.
What does that mean for the final answer?
Source: Qiu et al., Table 1, Sec 5.2.2. Tests built by the authors (App E).
The payoff
Better passages in, better answers out.
A fixed reader model got each tool's top five passages and answered MuSiQue questions. Exact match: 44.1% with the retrained tool's passages, 35.5% with Search-R1's.
exact match with Search-R1's passages
exact match with the retrained tool's passages
Our read: when you judge an AI search tool, ask how it was tested. A clean practice test can hide what it does after its own wrong turns.
Where does this stop working?
Source: Qiu et al., Table 2 (deepseek-flash reader, top 5 passages). The "Our read" line is interpretation.
The limits
Four-step questions are still mostly unsolved.
On four-step questions the retrained tool fully answers 6.1%. Every tool in the test stays under 7%.
All four steps in a row must go right, and one slip costs the rest.
How sure can we be of the rest?
Source: Qiu et al., Fig 2, Sec 5.2.3. Five-seed mean.
The limits
Read with care: one team, mostly one run.
- It is a preprint dated 29 October 2026. We found no outside replication.
- Most tests use one training run. Only the main table averages five.
- The authors built the long-chain tests, using language models and filters.
- Part of the gain comes from which wrong examples the tool trains against. Equalise those and the gain shrinks from 5.9 to 3.8 points.
- Training takes longer: 60 minutes against 37.
Where can you read it yourself?
Source: Qiu et al., Sec 5.1.3, App B (Table 5), App E, App F.2
Credit and sources
Sources and method
Credit
Research by Chenjie Qiu, Zixuan Zeng, Shaobin Shi, Bingfeng Chen, Dongsheng Luo and Boyan Xu.
Singapore Management University
Guangdong University of Technology, China
Paper: "Retrieving from Your Own Mistakes", arXiv 2610.40409v1, 29 October 2026. Illustrations drawn for this story. By vizmaya desk, 2 November 2026.
Method
Every number comes from the paper's text, tables or figures. Checks: 32.1 minus 20.8 = 11.3. 50.5 minus 44.6 = 5.9.