vizmaya desk · 3 Nov 2026

Two AI students. The same 32,768 practice tasks. One won all 12 comparisons.

Apple's SynthAPI study, explained from scratch.

12 of 12

What did the winner do differently?

Source: paper p9, Table 1 (p7). Win count computed from Table 1.

The job

An AI agent is a student who has to use apps.

You ask for something. The agent opens a shopping app or a music app and does the job in several steps.

Researchers call this tool use. It is hard because real practice runs are rare.

Where do you get millions of practice runs?

Source: paper p1. The three steps on the right are an illustration, not data.

  1. Read the request
  2. Pick a tool inside an app
  3. Check what happened, act again

The old rule

More practice makes a better student.

Most work on this problem chased volume. Make more examples, train on more examples, hope for more skill.

Does a bigger pile always help?

Source: paper p1 (prior work does "emphasize increasing data quantity").

Drawers of a library card catalog, from the Wikimedia Commons file Yale card catalog
Photo: "Yale card catalog", Wikimedia Commons. Author and licence on the file page.

The crack

A huge practice set left some students worse off.

Nemotron is a large existing tool-use dataset. After training on it, the models scored below their own starting point in 5 of 8 settings.

On BFCL, the 4B model fell from 27.3 to 15.9.

So what makes practice good?

Source: Table 1, p7. Differences and the 5-of-8 count computed.

The turn

Apple built a practice factory with four dials.

SynthAPI asks an AI to write fake apps, their databases, and tasks with answer checkers. A task that fails the checker is thrown out.

Then the team could change one dial at a time.

Which dial matters most?

Source: paper pp3 to 5.

  • AAppshow many different apps
  • TTaskshow many tasks per app
  • SStarting databasesdifferent worlds per task
  • DClutterextra, unrelated data in each world

The experiment

Then they trained 849 students and graded them.

Each student practised on a different dial setting. Then each one took tests on apps and tasks it had not seen.

What did the grades show?

Source: paper pp1, 5, 6. Four student sizes: 2B, 4B, 9B, 27B.

  • 849students trained
  • 458,752practice tasks made
  • 64fake apps
  • 10,472tools inside them

Evidence 1: apps

Small budget? Stay narrow. Big budget? Go wide.

With 64 to 512 practice tasks, 8 apps worked best. From 4,096 tasks, 64 apps was clearly best.

For the range in between, the paper's text does not say.

What about the starting databases?

Source: paper p7, Figure 2a.

Evidence 2: worlds

Same question, more starting worlds.

Keep a task the same and change the database it starts from. At 32 to 512 examples, one world per task was best. Past 4,096, four worlds per task won.

Put the dials together. Who wins?

Source: paper p8, Figure 2b.

The showdown

Same total. Different mix.

More tasks: 32 apps, 1,024 tasks each, 1 world, less clutter.

More worlds: 64 apps, 128 tasks each, 4 worlds, more clutter.

Both make 32,768 tasks. The first has 32,768 different questions. The second has 8,192.

Which student wins?

Source: paper p9 (called SubOpt and Opt there). Products computed.

The result

12 of 12

The "more worlds" recipe beat "more tasks" on every model size and every test. Gains ran from +0.3 to +7.0 points of success rate.

Could anyone have predicted this?

Source: Table 1, p7. Differences computed.

The forecast

Their formula saw the future coming.

They fit it on smaller runs, then asked it about runs up to 16 times bigger. It missed by 0.0285 on average.

A formula that only counts tasks missed by 0.1706. That is about 6 times worse.

So what do we do with this?

Source: pp2, 8. Ratio 0.1706 / 0.0285 = 5.99, computed.

The payoff

Next time someone brags about data size, ask about the mix.

  • How many different apps or worlds does it cover?
  • How many starting states does each task get?
  • How messy is each one?

Where does this stop being true?

Our reading of the result, not a claim in the paper. Evidence: slides 7 to 10.

A shopping cart in a supermarket, from the Wikimedia Commons file Shopping Cart Supermarket
Photo: "Shopping Cart Supermarket", Wikimedia Commons. Author and licence on the file page.

The limits

Bigger students gain less from the better mix.

On AppWorld, the gain shrank from +6.6 points at 4B to +0.3 at 27B.

On ASTRA-Bench at 27B, the better recipe scored 79.3. The starting model scored 80.2.

What else is missing?

Source: Tables 1 and 4, pp7, 23. Differences computed.

The limits

What is still unproven.

  • VerifiedThe authors tested the formula on one data pipeline. It may not transfer.
  • VerifiedNo simulated user. Agents never practise asking a human for details.
  • ContestedThe clutter gain may reflect tests that go easy on messy databases.
  • ContestedRival teams trained differently, so those comparisons do not prove the data is more efficient.

Will anyone repeat it?

Source: paper pp8, 9, 10. A preprint, not peer reviewed. OpenAlex showed 0 citations on 3 Nov 2026.

Credit and sources

Sources and method

Research by Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni, Michael Kirchhof, Cheng-Yu Hsieh, Raviteja Vemulapalli and Alexander Toshev at Apple. Explained by vizmaya desk.

Method: every number was matched to a page or table of the paper. Arithmetic is marked "computed". Byline: vizmaya desk, 3 Nov 2026.