Unmarked dark monolithic slab on a plain surface representing KV cache placement across a bus

A Factor of 22 on the Same Card: What KV Cache Placement Says About the Bus

Two placements that look like one flag

llama.cpp decides memory placement by layers. -ngl sets how many run on the GPU, and --fit picks that number when a model does not fit in VRAM. What the flag does not show is where the cache goes. A layer’s KV cache is allocated on model.dev_layer(il), so moving a layer to the CPU moves its cache to host RAM with it. Those are not the same cost. Weights in host memory are read once per token. The cache is read for every attention layer at every decoding step, and the attention pass waits on that read. A model 80 percent on the GPU is not 80 percent as fast. It crosses a bus on every step.

The measurement that separates the two

Unsloth’s placement planner exists because of that distinction. Its commit from August 26, 2026 (PR #9675, commit ba34b8f) spills tensors with -ot instead of layers so the cache stays on the GPU. The commit carries a measurement at 128K context on one B200:

  • Weights spilled, cache resident: 71.63 tokens per second.
  • Cache spilled, weights resident: 3.24 tokens per second.

Same card, same model, same memory budget: a factor of 22.1. It is the project’s own number, unreproduced by anyone else, so read it as a project measurement.

The mechanism is in the same commit. ggml moves an operation to the GPU only at batch size 32 or larger, and decode is batch 1. A tensor in host memory is read and computed on the CPU path, and the penalty tracks core count: the project’s cost model lists 2.42, 5.83, 11.82 and 14.94 tokens per second at 4, 16, 64 and 192 threads. The planner also refuses to plan on unified-memory hosts, because there spilling moves bytes between two names for the same chips. That is the clearest statement of what the penalty is: a bus, not a memory size. On a unified-memory machine there is no crossing, so the cliff does not exist.

The second measurement: the draft cache follows the wrong flag

Issue #28115, opened August 31, 2026, reports a narrower version of the same problem. When the KV cache is offloaded to RAM, the multi-token-prediction draft cache goes to RAM too and is copied back to VRAM on every draft step. PR #28117 proposed stopping that inheritance. The reporter’s numbers, from the server timings object, greedy, 200 tokens, fixed prompt, RTX 5090 with 5200 MT/s DDR5:

Prompt depth Before After Change Accepted draft length
65,536 tokens 19.2 t/s (399 ms per step) 27.2 t/s (281 ms per step) +41.7% 7.69, identical
122,880 tokens 13.3 t/s (621 ms per step) 21.1 t/s (393 ms per step) +58.6% 8.33, identical

Acceptance did not change at either depth. The draft proposed the same tokens at the same rate, so the whole gain is the removal of a per-step copy across the bus. That is the cleanest evidence that the cost is placement rather than model behaviour. The author argued the draft cache should have stayed on the GPU by default rather than behind a flag. Neither landed: PR #28117 was closed without merging, and the issue was closed as not planned on October 19, 2026 after going stale. The numbers are the reporter’s own.

What I measured on my own serving box

The serving box is a Ryzen 9 5900X with an RTX 3090, running Strata on a 125B mixture-of-experts model at IQ2_XS. The requests were sent from a different machine, a Ryzen 7 2700X, so wall-clock figures include a network round trip and are kept separate from the engine’s timings.

The engine keeps a fixed resident window per layer in VRAM and streams the rest from pinned host memory. That is a different placement model from -ngl: nothing moves wholesale to the CPU backend, only blocks cross. I ran three fresh-prompt passes at five depths with distinct filler text, so nothing came from the prompt cache, and took the median.

Prompt tokens read Decode median, 3 passes Range across passes Prefill median Draft acceptance
1,317 93.1 t/s 89.5 to 100.6 (12.4%) 1,102.9 t/s 77.8%
10,643 115.0 t/s 100.1 to 119.5 (19.4%) 2,143.9 t/s 76.9%
43,773 112.2 t/s 106.7 to 122.8 (15.1%) 2,364.2 t/s 86.4%
88,609 89.3 t/s 79.5 to 95.2 (19.7%) 2,372.0 t/s 72.0%
130,277 115.8 t/s 114.7 to 116.9 (1.9%) 2,320.7 t/s 80.0%

Decode did not fall with depth. It rose from the shallowest row, dipped at 88,609 tokens, and returned to its best value at 130,277. A placement cliff predicts a monotone fall once the working set passes the resident window. There is no such curve here.

The resident window is 32,768 of 131,072 cells per layer, so the deepest row put roughly 97,500 tokens’ worth of cache outside it. That row ran fastest, with the tightest spread of the five, 1.9 percent across three passes. The shallowest row was the slowest, the opposite of what a placement penalty predicts. The spread inside a single depth reaches 19.7 percent, the same order as the differences between depths. Depth is not the variable.

What the engine reports about the same placement

The startup log gives the numbers behind the flat curve. It probes the host-to-device path before loading and reports 26.4 GB/s as the best of four passes on every probe I read, with individual passes between 25.5 and 26.4. That is 82.5 percent of the 32 GB/s a PCIe 4.0 x16 link can carry.

Placement is reported by type, not by layer. The expert cache is 17.09 GiB of VRAM holding 12,754 slots ranked by routing frequency, with no eviction. The cache is 32,768 of 131,072 cells per layer in VRAM, with K and V in 1.55 GiB of pinned host memory. On a 24 GB card the engine reports 480 MiB free with everything loaded, and the prompt path borrows 3,208 slots from the expert cache.

Dividing reported cache traffic by reported miss count gives the per-block size. These are cumulative session counters, not measurements of my requests, so they describe the tier rather than a run:

Cumulative block reads Hit rate MiB read from RAM Bytes per block ns per block at 26.4 GB/s
11,843,292 95.70% 2,049.2 4,219 160
183,103,668 97.40% 19,154.4 4,219 160
222,780,732 97.42% 23,113.8 4,217 160
550,161,924 98.44% 34,503.3 4,216 160

Four snapshots taken at different points in the session give the same per-block size within four bytes. The same tier behaves differently at 4 KB than at 1 MB, which is the subject of the block size sweep I ran on the same class of drive. Once the working set passes the drive’s cache tier, the rate is set by queue depth, not the rated sequential figure.

The contrast is the point. The largest expert blob the engine reads is 1.51 MB, which is 57.2 microseconds per crossing at the probed rate. A cache block is 4,216 bytes, which is 160 nanoseconds of transfer time at the same rate. That 160 nanoseconds is arithmetic on the probed rate, not a measured latency: the log reports block counts and hit rates, not per-block time. What it does show is 23,114 MiB of cache blocks read in a session at a 97.4 percent hit rate. Rare crossings, each one small. A layer placement has no equivalent, because a layer on the CPU path is crossed once per token at the size of the layer. On the GPU side the ceiling is not the bus but the memory controller plateau on the base die.

What this means for a build

The practical decision is not how many layers to offload. It is which side of the bus the cache lives on, and on a discrete card that is usually decided for you by whatever loads the model. A wrapper that fills VRAM to the edge and then spills will move a layer, and its cache goes with it. InventiveHQ’s `-ngl` sweep on Qwen2.5-Coder-14B Q4_K_M, 48 layers, on an RTX 5060 Ti is the clearest published version of that: 2.89 tokens per second with every layer on the CPU, 12.5 at 40 of 48, 43.18 once the last eight move. That is a cliff, not a slope, and the last few layers carry most of it.

Three things change it. Keep the cache on the GPU and spill tensors instead of layers, which is what -ot does and what the planner in PR #9675 plans. Set the context you actually need before choosing a layer count: the cache competes with layers for the same VRAM, and a count that loads at 4K can fail at 32K. If the model does not fit, take the smaller quant that runs entirely on the GPU.

Host RAM is the part people underbuy. The cache that spills is pinned, not pageable: the engine calls cudaHostRegister with the portable flag and large pages. A 64 GB desktop kit is what holds a cache that does not fit in 24 GB of VRAM. The G.SKILL Trident Z5 64 GB (2×32 GB) DDR5-6000 CL30 U-DIMM kit is the form factor for a desktop board. I read the listing today at $1,349.99, which matches CamelCamelCamel’s October 30 scan. A SODIMM kit of the same capacity is laptop memory and the wrong part for a desktop board.

The storage tier matters for a different reason: a split engine reads the model table at load time, and this one reports loading 33.02 GiB at 2.82 GiB/s, about eleven and a half seconds before the first token. The WD_Black SN8100 2 TB is a PCIe 5.0 x4 M.2 2280 drive at about $529.99, which is CamelCamelCamel’s September 7 scan. Its listing was walled today, so that figure is the tracker’s. For a full set of quant files the TEAMGROUP MP44Q 4 TB read $499.99 on its listing on November 13, 2026 and was not re-readable today, cheaper per gigabyte and on PCIe 4.0. The full ladder is in best hardware for running local LLMs.

Decision rules

  • Read the placement, not the fit. A startup line that reports a CPU KV buffer tells you which side of the bus the cache is on.
  • Do not treat partial offload as a compromise. InventiveHQ’s sweep on a 48-layer model runs 2.89 to 12.5 tokens per second across the whole partial range, and the last eight layers take it to 43.18.
  • Keep the draft cache with the draft. The measured change was worth 41.7 percent at 65K depth and 58.6 percent at 122K with identical acceptance, and it was not merged.
  • Measure depth before believing a depth penalty. Five depths on a card with a resident window produced no monotone fall, and the deepest run was the fastest.

The factor of 22 is a real number on a real card, and it is the cost of putting the wrong thing on the wrong side of a bus. My own run does not show it, because nothing was placed wholesale on the CPU path. Small pieces crossed at 160 nanoseconds each, at hit rates between 95.7 and 98.4 percent. That is the difference between a placement policy and a layer count.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *