Janas: A 48 GB Model on 32 GB of RAM Says the Memory Wall Is a Cache Policy Problem

What happened

An inference engine called Janas ran a 48.4 GB model on a laptop with 32 GB of memory and produced 24 tokens per second. The same weights under llama.cpp on the same machine produced 2.8. The project, published under GPL-3.0 at github.com/prabanta-dev/janas, did not shrink the model. It changed where the model lives: weights stay on disk, and for each token the engine reads only the experts that token activates from an NVMe SSD. Experts that keep coming back stay in a RAM cache, which came to about 19 GB on the test machine. The rest was read from disk while the model answered.

The design decision is not the streaming. It is O_DIRECT. Reading with O_DIRECT bypasses the operating system’s page cache, so the engine decides what is in memory instead of the kernel. A kernel evicting a hot expert in the middle of a decode is what makes the naive version of this idea slow. That is the whole result, and it is a policy result, not a capacity result.

Why a 48 GB model fits in 32 GB at all

A mixture-of-experts model has a large total parameter count and a small active parameter count. Qwen3-Next-80B-A3B has 80 billion parameters in total and about 3 billion active per token. The A3B in the name is the working set. You do not need to hold the file in memory. You need to hold the slice a given conversation touches, and that slice clusters tightly enough that a 19 GB cache catches most of it.

That is a bet, not a guarantee. It holds while routing stays stable across a conversation and breaks when a prompt activates experts the cache has never seen, which is why reading a prompt is faster than generating. A dense model has no escape: a dense 80B needs all 48 GB per token, and no disk saves you. The technique is an MoE technique.

The project’s numbers

All figures below come from the project’s own repository, measured 9 to 12 October 2026 on one machine: Debian 13, kernel 6.12.107, Intel Core Ultra 9 185H, its integrated Arc GPU, 32 GB of DDR5, and a 1 TB NVMe SSD reading at about 5.7 GB/s. Q4_K_M weights, one round each, generation without draft models.

Read rates are tokens per second. The gap is largest exactly where the file exceeds memory and narrows to a normal margin on models that fit. That pattern is the evidence for the mechanism. On models that do not fit, llama.cpp’s usual build ran out of memory as soon as the GPU took part, because its CPU backend copies weights into its own memory to repack them. It ran only with repacking disabled and the experts memory-mapped, which is the page cache path. Janas ran the same files with no options.

Speculative decoding moves the generation column to 37 tokens per second on the 80B model. The project’s own docs report its multi-token prediction block taking that model from 23 to 36 tokens per second on an English explanation, and they are explicit that drafting changes how many tokens come out of one pass, never which ones. The author also states plainly that this is an experiment, that the API changes several times a week, and that everything was measured on one machine. Treat the ratios as a hypothesis about cache policy on one stack, not as a benchmark others have reproduced. If the reading gap reproduces on third-party hardware, NVMe expert streaming becomes a default technique for 32 GB-class machines. That is why I measured my own box.

What I measured on the machine that serves my models

One attribution point first: this measurement ran in my own process on a Ryzen 7 2700X host with 32 GB of memory, and the model was served by a different machine. The serving box is a Ryzen 9 5900X, 64 GB of DDR4, an RTX 3090, Windows 11, running a 125-billion-parameter MoE through llama.cpp behind llama-swap, with weights split across GPU, RAM and SSD. The full recipe for that build is in the Strata write-up. The throughput numbers below were measured through the API against that box. The disk numbers were measured on that box directly, against a 17 GB MoE quant on its model drive, a 2 TB SSD. I am not claiming that file is the one behind the model I run on; it is a file on the same drive, and the drive is what the measurement is about.

Three measurements, run today:

Two things fall out of this. The engine’s decode rate and the wall-clock rate through an API are different numbers, and the gap is request handling and tokenization. An agent loop experiences the wall-clock number: on this box the median is 202 against 141, a 30 percent gap. Any tokens-per-second figure compared across tools has to say which one it is. The second thing is the cold-read column. My model file is 17 GB and the serving machine has 64 GB of memory, so the file cache holds it and the disk path does not fire during a decode. The 1.35 GB/s average is what the device does on regions the cache had not touched; 4,960 MB/s is the same device on a region already cached. On a 32 GB laptop with a 48 GB file the cache cannot hold the file, so the disk path is real on every token.

The arithmetic that decides it

At Q4_K_M, about 3 billion active parameters is roughly 1.8 GB of weights per token. If every expert came off the disk cold, a 5.7 GB/s drive caps generation at about 3 tokens per second. Janas reports 24, which needs 43.2 GB/s of bytes per token. A 5.7 GB/s drive can supply at most 13 percent of that, so at least 87 percent of the per-token bytes must be served from cache. That is arithmetic from the project’s two published numbers, not a measurement, and it matters more than the token count.

Now put that hit rate on a drive that reads at 1.35 GB/s cold, which is what I measured on my own model drive. That is well below the 5 to 7 GB/s these drives are specified at, and the gap is the point: a spec sheet number is not what a cold read does. Same engine, same cache policy, same hit rate: 13 percent of 1.8 GB per token is 0.234 GB, and at 1.35 GB/s that is 5.8 tokens per second, stated as an estimate from the measured read bandwidth, not a measured token rate. The cache hit rate is the engine’s contribution. The drive is the floor under it when the cache misses. A machine that reads at 1.35 GB/s cold and a machine that reads at 5.7 GB/s are not the same experiment, and the published result belongs to the second one.

So the practical ceiling is not RAM capacity. It is whether the working set fits in whatever the engine can hold, and how fast the path behind it is when it does not. On my machine nothing streams, because everything fits. On the Janas laptop, 29 GB streams continuously and the engine still produced 24 tokens per second, because only 3 billion parameters are active per token. The memory wall is a policy problem, not a capacity problem, and the tooling has assumed capacity for years.

What to buy if you want to run models larger than your memory

Storage is the cheap half and it decides the floor. A drive that reads at 5 to 7 GB/s is the difference between 3 tokens per second cold and 0.7 on the drive I measured. The WD_Black SN7100 2TB is listed at $329.99 sold by Amazon as of October 18, 2026 (B0DN6ZQ3PD). A budget Gen4 2TB drive at $269.99 as of October 22, 2026, on a third-party listing with Amazon’s own out of stock (B0FPPVXVZK), is specified at 7,400 MB/s and does the same job. Plan for the whole file plus cache headroom.

Memory is the expensive half, because AI accelerators are consuming the wafer capacity that would have gone to desktop modules. A 64 GB DDR5 UDIMM desktop kit that sold for $123.99 in November 2023 is listed at $1,087 on a third-party listing as of October 16, 2026 (B0C79H54TQ). Same part number, same 288-pin UDIMM form factor, eight times the price. The mechanism is covered in what the memory shortage does to a local build, and the tiers worth buying now are in the hardware ladder from budget to enterprise.

The decision is therefore not “buy more RAM.” If the model you want runs at an acceptable rate with a working set that fits in the memory you already have, an engine that controls its own cache is the cheaper answer than a memory kit at ten times its 2023 price. A 32 GB machine plus a $270 drive is a working 80B MoE. A 64 GB kit at $1,087 buys the same model with the cache removed from the equation, and nothing else changes about the decode path.

Where this is still weak

Three things are not established. The results come from one machine and one author. The integrated GPU handles prefill, and a discrete GPU is untested, so the read-side numbers may not transfer to a box with a real card. And the cache is sized from free memory when the model opens, so anything else running takes from it: a 19 GB cache on a 32 GB laptop leaves 13 GB for the operating system. The README warns it can push a tight machine into swap. Every figure is one round, not a distribution.

What is established is the shape of the problem. A model file larger than memory is not automatically a model that cannot run. It is a model whose working set has to be managed by something, and the choice between the kernel’s eviction policy and the engine’s own cache is worth a factor of eight on generation. The drive speed sets what happens when that policy misses, and on a typical consumer drive that floor is about 1.2 GB/s, not 5.7. Both numbers are real. Only one of them is the interesting one.