vizmaya desk · 2 Nov 2026 · Alibaba research · arXiv 2610.39884
KARMA: an AI ad assistant got 29.7% better. Nobody retrained it.
Average score out of 100, on cases it had never seen. Before and after.
How do you improve an AI without touching the AI?
Source: paper abstract, p1; Table 3, p6.
Every day, a seller makes three choices.
Spend more or less on ads. Bid higher or lower. Add a product to the ads, or stop one.
In the study an AI agent made these calls from real shop data. It only suggested. No ad was changed.
How do you grade an AI on a choice like that?
Source: Eq 1, p3; p4.
Half a point for the move. Half for the direction.
Pick the right type of action and you earn 0.5. Get the direction right too and you earn 1.0. Pick the wrong type, or crash, and you earn 0.
"Right" means the same as what a real seller did in the logs.
So what grade did the AI start with?
Source: Eq 2, p4.
It started at 30.19 out of 100.
That is the average over three AI models and six product categories. Each was tested on 160 cases it had not seen.
A lot of room to improve.
What do you do with a weak AI?
Source: Table 3, p6; average computed.
The usual fixes all have a catch.
- Retrain the AI.Costly, and slow to repeat when products and rules keep changing.
- Rewrite the whole prompt.You cannot tell which change helped or hurt.
- Rewrite its code.Harder to test, harder to undo.
What if you changed none of them?
Source: p1; Figure 1, p2.
Freeze the AI. Edit only its rulebook.
The paper splits what the AI knows into six written parts: role, domain notes, experience rules, constraints, output rules and action descriptions.
Each rule has an ID, and each edit is logged before and after, so it can be traced and undone.
Who decides which rule to change?
Source: Section 4.3, p4; Table 1, p5.
A loop turns cases into a few small edits.
The AI answers 80 cases. A plain program sorts the wins from the failures.
A critic reads both piles and suggests changes. An editor makes at most 3 edits. The loop runs for 6 batches.
How do you know an edit helped?
Source: Figure 2, p3; Sections 4.4 to 4.5, pp4 to 5.
Every edit must pass a test, or it is undone.
The new rulebook runs on 160 fixed test cases. It stays only if the score rises by at least 1 point. Otherwise the old rulebook comes back.
One round on Electronics: 36.88 to 48.75, kept. Another: 41.25 to 34.38, undone.
Does it still work on cases it never saw?
Source: Eq 13, p5; Appendix D, p11.
On unseen cases, all three AIs scored higher.
Average: 30.19 before, 39.15 after. That is +8.96 points, or +29.7%.
These test cases were never used to pick an edit.
Did every category improve?
Source: Table 3, p6; averages computed.
16 of 18 settings improved. Two got worse.
A setting is one AI model on one product category. 16 of 18 went up.
Kimi-K3 lost 3.13 points on Automotive and 1.88 on Electronics. The biggest gain was +18.13, Qwen3.8-Max on Beauty and Health.
Which kind of evidence mattered most?
Source: Table 3, p6.
Wins teach as much as mistakes.
Study only failures: +8.23 points. Study only successes: +6.56. Study both: +16.67.
The two single channels add up to 14.79. Together they beat the sum by 1.88.
Tested on 3 categories, 1 model, 1 run each. On Beauty and Health it was a tie, 18.13 vs 18.12.
Can rules from one category help another?
Source: Table 4, p7; sums computed.
Rules learned in one category helped others.
30 transfer tests, no further edits: +8.54 points on average. That is about 71% of the +12.08 gain from a category's own rules.
Rules learned on Long-tail lifted Beauty and Health by 23.13, beating its own 18.13.
Was there a category that broke the pattern?
Source: Table 6, p7; Section 5.5, p8; averages computed.
Automotive broke the pattern.
Rules learned there lowered the score in all 5 other categories.
Inside Automotive, Qwen3.8-Max gained 6.88 on price updates and lost 3.13 on adding items and 3.75 on deleting them. The net was +0.63. A flat score can hide big moves.
So what can a reader take from this?
Source: Figure 3, p7; Sections 5.5 to 5.6, p8.
Try it: small edits, a fixed test, an undo.
- Write your rules down, one rule per line.
- Change at most 3 things at a time.
- Test on the same fixed cases. Keep a change only if it clears a bar you set first.
- Keep the old copy. Study what went right, not only what went wrong.
Our takeawayThis borrows the paper's method. The paper tested it on an AI agent, not on people.
Where does the evidence run out?
Source: method, pp4 to 5; evidence channel result, Table 4, p7.
Where the evidence runs out.
- Each main result is one run. A repeat test on Electronics gave gains from +0.94 to +11.25.
- The score measures agreement with past seller actions. It does not show more sales or profit. Nothing ran live.
- The data and code are not released, so nobody outside can re-run it.
- ContestedOur reading: the 1-point gate is about 3 case flips on 160 cases, so a small gain can pass.
Who did the work?
Source: pp4, 8, 9, 11, 12; Table 8, p11; gate arithmetic computed from Eq 2 and 3.
Credit and sources
Research by Xinhua Miao, Yuanzhen Rong, Weiru Zhang and Xiaoyi Zeng, Alibaba International Digital Commerce Group, Hangzhou, China.
- Paper: "KARMA: Training-Free Knowledge-Atomic Harness Evolution for Industrial Advertising Agents", arXiv:2610.39884v1, 29 Oct 2026. arxiv.org/abs/2610.39884
- Every number comes from the paper's tables and text, or is computed from them. Venue not confirmed, data private, no replication found yet.
- Not covered by the paper: a comparison with simple repeat-sampling baselines. One outside preprint argues such baselines matter (Contested): arXiv 2607.12227.
- Visuals are drawn for this story. More stories: vizmaya.fyi/stories.
vizmaya desk · 2 Nov 2026