What was actually measured
A KV cache that only one GPU can read is a cache that mostly misses. What is less settled is what happens when the router gives you nothing. That is the case Metrum AI tested in a single configuration, reported again through TrendForce on November 6.
The setup is four vLLM replicas, one per GPU, tensor-parallel size 1 each, identical engine arguments, all pointed at the same physical drive. The model is Qwen3-Coder-30B-A3B-Instruct-FP8, a 30B mixture-of-experts with 3B active. The host is a Gigabyte G293-Z23 with four RTX PRO 6000 Blackwell Server Edition cards, an AMD EPYC 9554 and 384 GB of DDR5. Storage is a Solidigm D7-PS1010 TLC drive, 15.36 TB, PCIe 5.0, with a Solidigm D5-P5336 QLC drive at 122.88 TB for the tier comparison. The router is plain round robin and is deliberately not prefix aware.
Two phases. Each document is sent once to populate the pool, then its follow-up goes to a replica that did not prefill it. Sixty-four prompts at roughly 99,000 tokens each, about 620 GB of KV state, roughly 10 GB per session. The page cache is dropped between every step, so every reload is a genuine device read. The only variable between the arms is one configuration value: private cache directories, or one shared directory.
| Metric | Private caches | Shared pool | Ratio |
|---|---|---|---|
| Reuse time to first token, p50 | 18.38 s | 2.20 s | 8.4x |
| Reuse time to first token, p95 | 19.5 s | 3.7 s | 5.3x |
| Reuse time to first token, p99 | 19.6 s | 3.8 s | 5.2x |
| Served warm within 3 s | 0% | 89% | n/a |
| 5 s SLO attainment | 0% | 100% | n/a |
| Wall clock for 64 requests | 338 s | 38 s | 8.8x |
| Mean GPU power during reuse | 413 W | 119 W | 3.5x |
| Errors | 0 | 0 | n/a |
Read the ratios as the source states them. The 8.4x compares medians. The 8.8x compares totals, and totals follow means, so a handful of slow recomputes in the isolated arm widen the second number. They are not the same measurement and should not be quoted as one.
Why the number is a routing number, not a storage number
The result is not that a shared directory produces cache hits, which is obvious on paper. It is that the test removed the mechanism everyone else relies on. Prefix-aware routing is the industry answer: Red Hat reports an 87.4% hit rate for llm-d routing across pods, and Baseten reports a 50% time-to-first-token reduction at an 89% hit rate across four replicas. A cache-aware router raises the odds the KV is already on the right GPU. It cannot raise them when a session moves because of autoscaling or failover, and it cannot raise them at all when the router is round robin.
With routing intelligence removed, the isolated arm is not a degraded case. It is the floor. Every request is a full recompute of roughly 99,000 tokens, so the distribution is flat: 18.4, 19.5 and 19.6 seconds across p50, p95 and p99, a spread of about 1.2 seconds. There is no second mode from partial hits, because there are none. Reload rate and SLO attainment are both zero. Those are not bad numbers, they are the absence of a mechanism.
The shared pool recovers what routing gave up. Hit rate stops decaying with replica count, because the cache is no longer attached to a replica. That is the structural change. The latency numbers are what it looks like on a request.
The tail is the part that decides whether it is usable
A median win with a broken tail is not worth shipping. Here the tail holds. p99 is 3.8 seconds against 19.6 seconds, 5.2x lower, and that p99 still sits below the isolated median. The shared arm clears a 5 second SLO on every request, against none in the isolated arm.
That is the difference between a cache that occasionally helps and a cache you can put a service-level objective on. Most storage benchmarks report the median and stop. The percentile spread is what an operator gets paged for.
The power figure is the one that scales
Mean GPU power across the reuse phase falls from 413 W to 119 W, a 3.5x reduction. The reason is mechanical: a 99,000-token prefill is dense floating-point work on every layer, while a KV reload is a device transfer with the GPU largely idle.
Energy per follow-up is the number that compounds. 413 W across 18.38 seconds is roughly 7.6 kJ. 119 W across 2.20 seconds is roughly 0.26 kJ. About 29 times less energy to serve the same turn, and the power ratio understates it because the shared arm finishes sooner. The 2.20 seconds is not free: it is roughly 10 GB moving off the drive and across PCIe. Below the crossover a recompute is cheaper than the read.
Same workload, two NAND tiers: the medians match
The tier comparison is the most transferable part of the report, because it is a drive choice a reader can actually make.
| Signal | TLC, Solidigm D7-PS1010 | QLC, Solidigm D5-P5336 |
|---|---|---|
| Drive read during reuse | 6.07 GB/s | 5.21 GB/s |
| Rated sequential read | 14.5 GB/s | 7.0 GB/s |
| Share of rated spec achieved | 41.9% | 74.4% |
| Average read I/O size | 972 KB | 123 KB |
| Queue depth, p95 | 45.7 | 585 |
| r_await p50 / p95 | 2.3 / 4.3 ms | 5.0 / 10.1 ms |
| Time to first token p50 / p95 | 11.4 / 15.3 s | 11.4 / 20.1 s |
| Composite temperature | 43 C | 34 C |
| Wear / available spare | 0% / 100% | 0% / 100% |
The percentages of rated spec are my own arithmetic from the figures printed in the same report, not published claims. The QLC drive ran at about 74% of its rated read rate, so the tail difference is not the drive running out of headroom. It is request shape. Smaller reads build a deeper queue, and a deeper queue moves the tail. That is the same mechanism as queue depth on a single drive, measured here across a fleet.
The practical reading is the one the source makes: TLC if you are judged on the tail, QLC if you are judged on capacity. The QLC drive holds eight times the storage, ran nine degrees cooler, and matched the TLC drive at the median. It lost about five seconds at p95. If your service-level objective is a median, the cheaper tier is fine. If it is a tail, it is not.
The ceiling is in software, not on the drive
Throughput saturated at about four concurrent reloads inside the LMCache fetch and copy path, not at the device. The drive figures are a floor rather than a limit, and the headline 2.20 seconds is the leftmost point of a concurrency sweep. At 16 concurrent reloads the shared arm reaches roughly 11.7 seconds, at 32 roughly 42 seconds. The headline describes a lightly loaded fleet.
Check the arithmetic against the measured transfer rate. Ten gigabytes at the flat 4 GB/s the drive held under load is 2.5 seconds, against a reported median of 2.20. The median is essentially the transfer time. The drive stayed flat at 4 GB/s against a measured ceiling of 13.9 GB/s, so the stage that bounded the request was the fetch path, not the NAND.
One dependency fails silently. Cross-replica hits require PYTHONHASHSEED=0 on every replica. Without it the shared pool returns nothing and nothing in the telemetry says why. A shared pool is a storage tier, not a memory tier, and the distinction matters: the CXL tier is not faster and not cheaper, because the slot wall and the price per gigabyte both move against you. A shared directory on a drive is cheaper and slower, and it is honest about being slower.
What this means on one machine at home
There is no cross-replica problem on a single GPU, so the 8.4x does not transfer directly. What transfers is the tier logic. A local agent loop that keeps a repository context across sessions is the same workload in miniature, decided by two facts: block size, and whether the cache is readable by the process that needs it.
At a few thousand tokens a recompute is cheaper than a read and you are not storage-bound at all. The crossover is what makes the disk tier worth having. Block-size behaviour on the same class of drive is measured in what block size says about the KV cache tier, which explains why a consumer drive reading at 2,349 MB/s in a large block can read at 218 MB/s in a 4 KB one. A KV reload is not a large block. It is a stream of medium blocks through a fetch path that saturates at four concurrent readers.
For a single-machine build the memory ladder is already in the guide to hardware for running local LLMs. This report adds one decision: which tier you buy for the cache. If your contexts are long and your sessions repeat, a drive sized for the cache is not a storage purchase. It is a memory purchase that happens to be slow.
What to buy for the cache tier
Prices below are read from the live listings on November 12, 2026, and are approximate. The first figure is the 2 TB listing, the others are 4 TB.
For the tail, a PCIe 5.0 TLC drive is the tier that held p95 five seconds lower in the measured run. The WD_Black SN8100 2TB lists at about $479.99, roughly $0.23 per gigabyte. That is a poor capacity buy and a good latency buy, and the report is explicit about which one it is.
For capacity, QLC is the tier that matched at the median and ran cooler. The Sandisk Optimus 5100 4TB is a QLC part on a PCIe 4.0 link at about $599.99, roughly $0.15 per gigabyte. The TEAMGROUP MP44Q 4TB is the same class at about $499.99, roughly $0.12 per gigabyte, and is the cheaper way to hold a pool of sessions rather than a fast path to read them.
Neither is the measured drive. The D7-PS1010 and D5-P5336 in the test are enterprise parts, and the gap between them and a consumer drive is not the NAND. It is queue behaviour under load. A drive at 74% of its rated read rate is a drive being fed small reads by a pipeline that cannot keep up, which is the software ceiling moved one layer down.
What bounds these numbers
This is a preliminary technical evaluation from a single test configuration, reported as observed behaviour rather than as a guarantee. One server, one model, one router policy, two drive tiers, concurrency 1 for the headline. No multi-tenant contention was tested, which is where the GPU power saving would matter most. The isolated arm under load was not measured, so the comparison at load is not defended by data.
That is the honest boundary. Inside it, the shared tier served 89% of requests warm within three seconds, met a five-second SLO on all of them, and did it at 3.5x less GPU power, on a drive that never became the bottleneck. Any one of those could have failed in practice, and any one would have ended the idea.

