A plain matte black storage module on a dark surface

The Same SSD Reads at 218 MB/s or 2,349 MB/s: What Block Size Says About the KV Cache Tier

The same drive, two numbers

The drive in my machine is a Patriot P320 256GB built on a Maxio MAP1202 controller, a DRAM-less PCIe 3.0 x4 part with four NAND channels and no memory interface. I read 256 MB of it under O_DIRECT at six block sizes, five passes each, so the page cache could not flatter the numbers. The result depends entirely on how large a piece the reader asks for.

Read size Measured (median of 5) Share of the 3.94 GB/s Gen3 x4 channel Time to fetch one 28 MB object
4 KB 218 MB/s 5.5 percent 134.7 ms
16 KB 589 MB/s 14.9 percent 49.9 ms
80 KB 1,332 MB/s 33.8 percent 22.0 ms
128 KB 1,456 MB/s 37.0 percent 20.2 ms
256 KB 1,588 MB/s 40.3 percent 18.5 ms
1 MB 2,349 MB/s 59.6 percent 12.5 ms

Same drive, same controller, same NAND, same host. The last column is the one that matters for a KV cache tier: the identical object takes about eleven times longer to fetch when the storage layer asks for 4 KB pieces instead of megabyte pieces.

The 28 MB object is not hypothetical. The py-kvcache characterization reports that a 256-token block for Llama 3.2 3B occupies about 28 MB on disk, one file per block, because that model carries 112 KiB of KV data per token. That is the unit a serving stack actually reads.

Where the cost sits

Per-read latency, measured separately over 2,000 reads at each size:

Read size Mean per read p99 Transfer time on the lane alone
4 KB 31.1 microseconds 72.1 microseconds 1.0 microsecond
80 KB 63.4 microseconds 113.8 microseconds 20.8 microseconds
1 MB 594.5 microseconds 890.8 microseconds 266 microseconds

At 4 KB, data movement is about 1 microsecond at the lane rate (1.7 microseconds at the rate this drive sustains at 1 MB) of a 31 microsecond request. The rest is command processing, mapping lookup, and the syscall round trip, and that fixed cost does not shrink with the block. A 28 MB object in 4 KB pieces is 7,168 requests. Read as one request, it is one.

One honesty note. The throughput loop overlaps request submission; the latency loop does not. At 4 KB the effective per-request cost sits between 19 and 31 microseconds depending on how the reader is written. Both point at the same thing.

Three stacks, three granularities

This is not a benchmark curiosity. Current KV offload systems differ mainly in the size of the object they ask the device for, and the papers describing them report the same fault line at different scales.

ScoutAttention (DAC 2026) states the arithmetic plainly: about 4 KB of KV data per token per layer over a PCIe 4 x16 link gives an effective I/O bandwidth of about 800 MB/s. A 32-token page of 128 KB raises it to about 15 GB/s. The granularity, not the device, is the constraint.

LMCache is the layer most people actually deploy. The October 2026 block-layer characterization of it traced two deployment snapshots with bpftrace on tracepoint:block:block_rq_issue. The older path (0.3.0, buffered I/O, page cache) showed 0 to 0.05 MB of reads at the block layer across four workloads. The newer path (0.3.16, prefix-aware dedup, O_DIRECT) collapsed writes on an exact-repeat workload from 1.62 GB to 11.75 MB, and then read that same 11.75 MB back on every warm request: 95 reads per request, 9,405 reads across 99 requests. Warm time to first token went from 27 ms to 93.1 ms.

The mechanism is a granularity mismatch. vLLM’s GPU prefix cache works at 16-token blocks; LMCache persists 256-token chunk files. A 94-token request leaves a 14-token tail that does not fill a block, so the whole partial chunk file is read to recover a tail that would have been cheaper to recompute. The paper measures that as about 6.7x read amplification.

Tutti takes the opposite position: keep the granularity the engine already uses and make the device tolerate it. For a 64-layer Qwen3-32B model at 128K context that is about 256,000 scattered 80 KB objects, roughly 21 GB. On my drive at the 80 KB rate that is 15.8 seconds of read time; at 1 MB granularity, 8.9 seconds. The paper notes that writes for the previous layer and reads for the next contend for the drive’s internal cache, so my read-only figure is a lower bound.

Reproducing both patterns on my drive

I ran the two I/O patterns the LMCache paper describes against the same drive.

The warm path: 95 reads of about 124 KB, 11.75 MB total, in 12.0 milliseconds. That is 12.9 percent of the 93.1 ms warm TTFT the paper reports. The other 80 milliseconds is the application and kernel path, not the device.

The cold path: the paper’s flush_ram experiment evicted the page cache and re-read the same data, producing 8,002 128 KB read-ahead events, about 1 GB in 683 ms, roughly 1.43 GB/s. I replayed it: 8,002 reads of 128 KB in 808 milliseconds, 1,330 MB/s. That is 93 percent of the paper’s rate on a Crucial P310, a Gen4 drive with a Phison E27T controller, achieved on a Gen3 drive with no memory chip on the PCB.

At 128 KB granularity the missing DRAM barely matters. At 4 KB it matters more, because that is where the controller has to look up scattered objects instead of streaming a run. I cannot separate the two effects on a single drive, so I am stating the boundary rather than the cause.

Why this is a policy decision before it is a hardware decision

The device number is not the decision. The decision is whether the object the storage layer hands back is smaller than the work it was meant to save. A 14-token tail costs microseconds to recompute and 12 milliseconds to fetch. That is the same argument the memory wall being a cache policy problem makes at the RAM tier, moved one tier down.

The buffered path looks read-free only while the page cache is warm. That cache is host memory, and on a machine running a model, host memory is the resource the model wants. It is also the pool this drive borrows its 8 MB Host Memory Buffer from, and the resource under pressure in the DRAM price increase. A cost model that assumes RAM residency can be broken by anything else on the machine.

For a local build the practical reading is narrow. If your workload is repeated prompts and you have memory slack, buffered I/O still wins on average, and it wins because the reads never reach the device. If you need a stable storage budget, O_DIRECT exposes the true cost upfront, and the number to set expectations against is the small-block rate, not the number on the box. This drive is rated at 2,200 MB/s sequential read. I measured 218 MB/s at 4 KB single-flow, 9.9 percent of the rated figure, and 760 MB/s at 32 concurrent flows, 34.6 percent. The rated number is reachable at 1 MB granularity, where I measured 2,349 MB/s, but not by a single-threaded loop reading in 4 KB pieces.

The queue-depth behaviour of this drive is covered separately; the short version is that four concurrent readers produced less than one reader at the sizes a single-threaded loop produces. Granularity is the lever that does move: 218 MB/s at 4 KB against 2,349 MB/s at 1 MB on the same hardware is the largest single change available without buying anything.

What to buy, if you buy anything

Storage is the cheapest tier to change and the one with the least effect, so the buying advice here is small.

  • If the cache stays in RAM, the drive does not matter. A 28 MB object read at 2,349 MB/s is 12.5 milliseconds. The same bytes from host DRAM are a fraction of a millisecond, which is arithmetic, not a measurement I took here. Buy memory before a second drive. A 16 GB DDR4 kit read $129.99 on the live listing on October 31, 2026, against a $26.99 low in February 2025.
  • If you do run a disk tier, the DRAM question is a small-object question. The Solidigm P44 Pro 1TB carries its own mapping-table memory and listed at $163.99 on October 31, 2026. The Crucial P310 2TB is DRAM-less and borrows 64 MB from the host, listed at $308.95 on the same day. At 128 KB granularity my DRAM-less drive matched 93 percent of the rate the LMCache paper measured on a P310. The difference between those two drives shows up in scattered small objects, not in a sequential scan.
  • Change the granularity before changing the hardware. Reading a whole chunk file to recover a 14-token tail costs 6.7 times the read needed for the tail itself. That is a layout choice, and it is free.

For where a storage tier fits against GPU memory in a full build, see the hardware ladder from budget to enterprise.

What I could not measure

Two boundaries worth stating. The first is a correction to an earlier measurement: the 51.5 percent figure was computed against the Gen3 x4 channel as an assumption. It is now confirmed: /sys/bus/pci/devices/0000:0c:00.0 reports 8.0 GT/s at x4, so the 3.94 GB/s denominator above is the actual link, not a guess.

The second is that every number here is a single-flow read on one drive, on one host, on a 256 MB file. No writes were in flight, so there was no read-write contention for the controller’s internal cache, which is the condition the Tutti paper identifies as the real degradation. The test file was written fresh, so it likely sits inside the drive’s 40 GB pSLC cache, which is why the 1 MB figure clears the rated number. A live serving loop reading and writing at once will be worse. What is measured is the shape of the curve, and the shape is the useful part.

7 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *