An AMD Fiji graphics chip package: the GPU chip sits beside stacks of memory on a flat carrier.
Photo: AMD Fiji GPU package, C. Spille / pcgameshardware.de, Wikimedia Commons, CC BY-SA 4.0

vizmaya desk · 4 Nov 2026

GPU utilization: what “busy” really means

Your dashboard says the chip is busy. Busy doing what?

A 15-page paper tries to find out. Start with one word.

Source: Shaikh, “GHOST-Driver”, p.1

A high number feels like a full tank.

nvidia-smi is NVIDIA’s standard tool for watching a GPU. It shows one utilization number per chip.

It looks like a fuel gauge. High means the chip is working hard.

A graphics card photographed up close, with its circuit board and components visible.
Photo: A Complex Graphics Card, Nick Stathas, Wikimedia Commons, CC BY-SA 4.0. Illustration, not from the paper.

Is that really what the number measures?

Source: paper p.1-2

The number asks one yes-or-no question.

Was any task running on the chip during this short window? If one task ran the whole time, the meter reads fully busy.

That holds even when the task uses one tiny corner of the chip.

Verified

So what happens when a job is stuck waiting?

Sources: paper p.1-2; PyTorch docs; NVIDIA forum; Netdata guide. Grid is an illustration.

Waiting does not count as busy.

The paper says driver delays hide inside the busy number. NVIDIA defines the number as time with a task running.

A job stuck waiting is not running. By that definition, it would not count.

Contested

And how long is the window the number is averaged over?

Sources: paper p.1 (claim); NVIDIA definition via PyTorch docs. Strip is an illustration, not to scale.

Even the clock is in dispute.

The paper says nvidia-smi samples every 100 ms. NVIDIA’s own wording gives a period between 1 second and 1/6 of a second.

1/6 of a second is 166.7 ms, so even the fastest case is slower than the paper says.

Contested

If the number is blurry, what is really going on inside?

Sources: paper p.2, p.4, p.9; NVIDIA forum; PyTorch docs. 166.7 = 1000 / 6 (computed).

A job takes a hidden road before it reaches the chip.

Your app does not talk to the chip directly. Its work first passes through the operating system’s GPU driver, which checks memory, waits for earlier work and queues the job.

The paper says normal profiling tools cannot see inside that road.

Rows of network cables plugged into equipment in a server room.
Photo: Network cables in server room, ProjectManhattan, Wikimedia Commons, CC BY-SA 3.0. Illustration, not from the paper.

What if you could time every stop on that road?

Source: paper p.1-3

The paper puts six stopwatches on the road.

The stops are: request arrives, memory check, wait for earlier work, handed to the queue, chip scheduled, done signal.

It stamps the time at each stop and adds them up for every job.

Which stop does the paper bet is the slowest?

Source: paper p.6-7, Table I (small labels are the paper’s stage names)

The paper bets the waiting stop is the slow one.

That stop is a fence wait: a job waits for earlier work to finish. The paper calls it the main blocking point, citing Linux kernel documents.

It drew the ranking as a sketch. It did not time it.

Sketch, not measured

How do you watch every job without slowing it down?

Source: paper p.7 (Stage 3), p.9-10, Fig. 3. “No absolute values are claimed”, p.10.

Tiny watchers can note the time without changing the driver.

The tool uses eBPF, a Linux feature that runs small, safety-checked programs inside the operating system.

Each watcher fires when a job passes one stop and writes down the time.

What does all that watching cost?

Source: paper p.2-3, p.5-8

5,000 jobs a second cost 2.655 ms of CPU.

2.655 ms vs 20 ms

That is the paper’s own math: 3 × 78 ns plus 3 × 99 ns is 531 ns per job, times 5,000 jobs. The paper rounds it to 2.7 ms against a budget of 20 ms, which is 2% of one core.

Computed

What if jobs arrive faster?

Source: paper p.9 (inputs and budget); arithmetic is ours. 531 × 5,000 = 2,655,000 ns = 2.655 ms.

The watchers alone hit the budget near 37,665 jobs a second.

Cost grows in a straight line: 531 ns for every job. Double the load and the cost doubles.

This counts the watchers only. It leaves out the maps, the buffer and the daemon that the tool also needs.

Computed

So what should you do with all this?

Source: paper p.7-9 (stage mapping, costs, 20 ms budget); arithmetic is ours: jobs per second × 531 ns. 20,000,000 / 531 = 37,665.

Next time a GPU looks “busy”, ask three questions.

  1. How long is the window it averages over?NVIDIA wording: 1/6 s to 1 s
  2. Busy with how much of the chip?The number does not say
  3. What is each job waiting on before it runs?The question this paper wants answered
A computer workstation with its lid off, showing the parts inside.
Photo: GPU workstation lid off, Wikideas1, Wikimedia Commons, CC0. Illustration, not from the paper.

Does the paper’s tool answer the third question yet?

Sources: slides 3 to 5; paper p.1-2. Window range is Contested (slide 5).

The tool has no hardware results yet.

The paper says so itself: its overhead is arithmetic, its delay chart is a sketch, and its gains are projections. Its own numbers also disagree in places.

0

hardware measurements. “Empirical validation… is in preparation”, p.14.

12 vs 63-99

nanoseconds per event. Same paper: p.2 says 12, p.9 uses 63 to 99.

56% vs 74.7%

the paper’s projected combined cut, p.12, vs what its four parts would give if they stacked. Computed. No method shown.

And will its tripwires survive the next Linux update?

Source: paper p.2, p.9, p.10, p.12, p.14. 74.7% = 1 − (0.58 × 0.74 × 0.83 × 0.71), computed. The 56% is a projection, not a measurement.

Its road signs use names that changed between kernels.

The paper says one binary works from Linux 5.10 to 6.x. A newer kernel header lists the scheduler’s trace events under new names.

I read header snippets, not the full files, so treat this as a flag.

Contested
KernelTrace event names seen
5.10.46
matches the paper		drm_sched_job_wait_depdrm_run_jobdrm_sched_process_job
v6.19-rc7
newer		drm_sched_job_queuedrm_sched_job_rundrm_sched_job_add_depdrm_sched_job_done

Has anyone measured something like this for real?

Sources: paper p.7, p.12; 5.10.46 header; v6.19-rc7 header.

A related tool, gpu_ext, did measure real workloads.

4.8×best-case throughput gain
2×best-case tail latency cut

These are the gpu_ext authors’ reported results from real tests, with low overhead for instrumentation only. It is a different system with a different goal.

Its numbers are not comparable with this paper’s projections.

Verified: arXiv abstract

Where to read more, and who to credit.

Source: gpu_ext, arXiv:2512.12615; paper p.13

Sources & method

The paper

  • Azhar Shaikh, “GHOST-Driver: A Non-Intrusive eBPF Framework for GPU Kernel Driver Latency Observability in AI Inference Infrastructure”, 2026, 15 pages. Attached PDF; file name 2611.03649v1.

Checks

Method

  • Every number is from a paper page or computed, with inputs shown. Contested claims are labeled. Claims I could not check, such as 47%, 200 µs, 3.7× and the projected gains as results, are left out. I did not open the paper’s references.

Photos

Paper by Azhar Shaikh. Research and story by vizmaya desk, 4 Nov 2026. More vizmaya stories

Source: full list above