One wall inside an AI chip capped chats at 34. Take it down and you get 90.

The room

An AI server keeps two things in one small room.

One is the model: the numbers it learned.

The other is notes on every chat running right now.

The room is the memory on a graphics chip, or GPU. It is fast, costly and small.

How do the two share the room?

Source: Fang et al., p1 and Fig 1, p2.

The old world

Today a wall splits the room. It is set at start-up.

Model on one side. Chat notes on the other.

The paper calls it a predefined fixed boundary. Once the server starts, the wall does not move.

Illustration, not data

What if the model is huge?

Source: p2, Fig 1(b).

The crack

A big model leaves little room for notes.

The authors' example: a model called DeepSeek-V4-Flash on eight A100 cards leaves about 7 GB per card for chat notes.

That is enough for just two chats of 1 million tokens each. A token is a small chunk of text.

The authors' own figure

Can the model take up less room?

Source: p1. Not checked outside the paper.

First fix

Park the spare parts somewhere else.

A big model is a team of specialists, called experts. For each word, only a few of them work.

So keep the busy experts on the chip. Park the rest in the computer's main memory.

34 to 74chats at once, on the same card

Is that the end of the story?

Source: p1, Fig 4(a) and 4(b), p4.

The wall is still there

A fixed wall wastes room in both directions.

Busy hour: the notes side fills up and chats wait, while the model side holds experts nobody is using.

Quiet hour: the notes side sits half empty and cannot keep more experts.

Illustration, not data

What if the wall could move?

Source: p2 and p4.

The turn

Take the wall down. Share one pool.

Cut the room into equal pages. Any page can hold an expert or chat notes.

In a busy hour, experts hand pages to chats. In a quiet hour, they take them back.

Illustration, not data

But experts and notes come in different sizes.

Source: p2, Fig 1(c). p6, Fig 6.

How it works, part 1

Pages come in three sizes, like a parking lot.

A huge page splits into medium pages for experts and small pages for chat notes.

When they are freed, they merge back, so no space is stranded.

Drawing is an illustration

Experts now sit in scattered pages. Can the chip still use them?

Source: p6 and p7, Fig 7.

How it works, part 2

Read each expert where it sits.

Old speed code wants experts packed in one block. Packing means copying, and copying costs time.

The new code follows a map of pages and reads in place. The extra cost: 0.83% on average.

Who decides when the wall moves?

Source: p7, p12, Fig 12.

How it works, part 3

A thermostat moves the boundary.

Free expert pages below 2: shrink the expert cache. Above 9: grow it. In between: leave it alone.

The gap stops it from flipping back and forth.

Curve is an illustration. The 2 and 9 are from the paper

Does it pay off?

Source: p8, Algorithm 1.

The result

On one card: 90 chats at once.

Same model, same card, same requests. Peak chats: 34, then 74, then 90. Steps to finish: about 2,100, then 1,300, then 1,000.

Contested: the paper says sharing cut steps by about 30%. Its own numbers give 23.1% (1,300 to 1,000, computed).

Does it hold in the full tests?

Source: Fig 4, p4 and p5. p2. Model: Qwen1.5-MoE-14B on one RTX 5090.

The full tests

Up to 19.03×, with a catch.

Bars show Rimuru's best result against each rival, in output per second.

The authors say the 19.03× rival ran with a weaker CPU path. Against KTransformers on a machine with a fast CPU path, the best case is 2.30×.

What does Rimuru give up?

Source: p10 and p11, Figs 9 and 10. All values are best cases.

The price

Each reply is slower.

Rimuru serves more chats at once, so each chat waits longer per word. The authors say so.

Bars are read off Fig 10 by eye, so treat them as about right, within 10%.

So what do you do with this?

Source: p11, Fig 10 (read from figure).

The payoff

What this changes for you.

The paper's core idea: memory should follow demand, not a split made at start-up.

Where could this break?

Source: p2, p9, p10. arXiv listing.

The limits

Five days old. Nobody has re-run it.

The numbers above are the authors' own, from tests they designed.

Sources and credit are next.

Source: p9 to p11, p13. arXiv listing.