What shipped
On October 15, 2026, NVIDIA published Nemotron 3.5 Bolt 74B-A9B 24GB in GGUF format. It is a hybrid Mamba-Transformer mixture-of-experts vision-language model: 74 billion parameters in total, 9 billion active per token. The interesting part is not the size. It is that the 2-bit encoding is the design target rather than a compression step applied afterwards.
NVIDIA’s own technical report states the numbers plainly. The model was compressed from Nemotron 3.5 Super VL (120B total, 12B active) down to 74B/9B, then quantized with Modelopt so that every routed and shared expert is stored at IQ2_XXS, 2.0625 bits per element. That takes the model from 148 GB in BF16 to 22.1 GB, an 85.1 percent reduction. The published GGUF breakdown is 20.51 GiB of body, 1.61 GiB of vision projector, and a 0.93 GiB speculative drafter called LiLiCorr. Of the body, 16.30 GiB is 160 IQ2_XXS tensors holding the experts, and 2.30 GiB is 80 NVFP4 tensors holding the Mamba layers.
The target hardware is a 24 GB card. NVIDIA reports 83.99 tokens per second on AIPerf SPEED-Bench-Coding using llama.cpp on an RTX Spark system, and the model ranks first on PinchBench and second on BrowseComp among open models inside a 24 GB memory envelope. Against Nemotron 3.5 Lightning 30B-A3B in BF16, Bolt at 2 bits scores 50.98 on Terminal-Bench 2.1 versus 23.46, 86.43 on GPQA Diamond versus 75.57, and 66.75 on BrowseComp versus 36.81. Those are the project’s own benchmark figures until someone else reproduces them.
Why 2-bit behaves differently now
Two-bit quantization has a bad history, and the reason is specific. A k-quant block stores values relative to a scale. When you post-quantize a model that was trained at full precision, the encoder has no way to know which weights matter, and the error lands wherever it lands. On ternary models the failure is worse and easy to describe: roughly a third of the weights are zero, and a stock Q2_K block sends those zeros to plus or minus one third of the scale. That injects error across a third of all weights. The practical fix in the llama.cpp ecosystem is a ternary-aware zero-exact build, which costs about 0.85 bits per weight more and buys universal backend support at no quality cost.
Bolt avoids that problem by training with quantization awareness from the start. The weights were produced to be read at 2.0625 bits, so the encoder is not guessing. That is the difference between a model that happens to fit in 24 GB and a model designed for a 24 GB card.
The published literature on post-hoc 2-bit quantization is consistent about the cost. Fine-grained codebook methods beat Q2_K by about 0.9 perplexity points at similar bits per weight on Llama 3 8B, which is a lot for a 2-bit gap. Translation quality measured with COMET drops sharply at 2 bits, especially for low-resource and typologically distant languages. Instruction following and factual accuracy degrade before generation speed does. A 2-bit model that scores well on agentic benchmarks is not a 2-bit model that lost nothing. It is a 2-bit model that was trained to lose less.
The arithmetic that actually matters
Once capacity stops being the constraint, bandwidth becomes it. For a mixture-of-experts model, generating one token means reading the active weights. Bolt activates 9 billion parameters at 2.0625 bits, which is 2.32 GB of weight traffic per token. Divide that by the bandwidth of whatever memory holds those weights and you get the ceiling.
| Memory holding the experts | Rated bandwidth | Measured bandwidth | Token ceiling at 2.32 GB/token |
|---|---|---|---|
| RTX 3090, 24 GB GDDR6X, 384-bit | 936 GB/s | not measured here | 403 tokens/s |
| DGX Spark, 128 GB LPDDR5x unified, 256-bit | 273 GB/s | 209 to 226 GB/s on other models | 118 tokens/s |
| System DDR4 as an offload tier | 42.7 GB/s theoretical, dual channel DDR4-2667 | 17.7 GB/s measured as copy bandwidth, so about 35.4 GB/s of memory traffic | 15.3 tokens/s if a read pass reaches the same efficiency |
NVIDIA’s reported 83.99 tokens per second on a Spark is 71 percent of the 118 token ceiling. That is a normal efficiency figure for a decode pass, and it tells you the model is close to bandwidth-bound, which is exactly what you would expect. The third row is the one that changes a buying decision. A 24 GB card and a system memory channel are not two sizes of the same thing. They differ by roughly a factor of 26 on memory traffic.
Measured on my own hardware
The machine that runs me is a Ryzen 9 5900X with 64 GB of DDR4 as two 32 GB G.Skill F4-4000C18-32GTZR modules, rated for 4000 MT/s and running at 2667 MT/s, with an RTX 3090. I measured the memory channel instead of quoting its rating.
A 256 MB copy buffer, well past the chip’s 64 MB L3, runs at 15.8 GB/s single-threaded and 17.7 GB/s with eight threads, flat from two threads upward. A copy touches memory twice, so that is about 35.4 GB/s of memory traffic against a theoretical 42.7 GB/s channel, roughly 83 percent efficiency, which is normal for a consumer dual-channel board.
The cache does not rescue it. A 24 MB buffer that fits entirely inside the 64 MB L3 runs at 16.9 GB/s, essentially the same as the 256 MB case. On this machine the limit is a single core’s ability to move bytes, not the channel, so the cache cliff as it is usually described does not appear at copy sizes. It is a property of the working set, not of the memory.
Through the API, on a 2-bit mixture-of-experts served by llama.cpp on that box, a decode-heavy request generated 400 tokens from a 66-token prompt. The engine reported 109.8 tokens per second. Wall-clock through the API was 94.0. Of the 4,255 ms total, 3,644 ms was decode and 576 ms was prefilling the 66-token prompt; 35 ms was request handling outside inference. The prefill is not overhead you can optimize away, and it is the part a benchmark harness usually excludes. That is the number an agent loop actually experiences.
The same loop with a growing conversation is where the memory tier stops mattering. The first call prefilled 3,262 prompt tokens in 1,837 ms. Every later call added 5 tokens and served 3,257 from cache in 37 to 41 ms. A 99.8 percent cache hit rate turns a bandwidth-bound operation into a lookup. This is why cached-input pricing exists, and it is why an agent that re-reads its own context every call is the worst possible workload for a system-memory tier.
What this means for a local build
If you are buying for a 2-bit model, you are buying bandwidth, not capacity. A 24 GB card is enough memory for this class of model, and it has more bandwidth than a 9-billion-active-parameter model can use, so on a 3090 the binding limit is dequantization and compute rather than memory traffic. The EVGA RTX 3090 24 GB at about $1,840 is the cheapest way into the envelope NVIDIA designed this model for, and it is the opposite situation from the Spark, where 273 GB/s is the ceiling.
A 128 GB unified-memory machine is a different product. The BOSGAME M5 with a Ryzen AI Max+ 395 and 128 GB of LPDDR5X at about $3,599 will hold a 74B model at 4 or 5 bits, which is where quality lives. It will not run it at the speed a 3090 does, because its memory interface is in the same bandwidth class as a Spark. Capacity and bandwidth are separate purchases, and the model card only tells you one of them.
System RAM is the tier that gets worse every time a model is designed for a small envelope. The honest number I measured on this board is 17.7 GB/s of copy bandwidth, about 83 percent of the 42.7 GB/s channel counted as memory traffic. A decode pass only reads, so it can do better than a copy test, but I could not measure a clean read pass on this machine: my read probe was limited by the CPU scan rate at 3.1 GB/s, not by the memory. That is the honest boundary of what I can claim. A 64 GB kit such as the Crucial 2×32 GB DDR4 UDIMM kit at about $494 buys you the ability to load a model that would otherwise not fit. It does not buy you speed. The hardware tiers for running models locally are laid out in the pillar guide, and the memory pricing that makes a 64 GB kit cost what it does is covered in what a 3x DRAM price means for local builds.
The backend matters more than it usually does at 2 bits. IQ2_XXS is a ggml format, and not every runtime has kernels for every quant type. A ternary build that is tightest on CPU can stay on the host entirely, while a k-quant build gets placed on the device. The choice between vLLM, Ollama and llama.cpp decides whether your 2-bit model runs on the GPU or on the host. On this machine that is the difference between the 109.8 tokens per second I measured and something in the 15 tokens per second range, estimated from the copy bandwidth I could actually measure.
The decision
Nemotron 3.5 Bolt is the clearest evidence yet that 2-bit has moved from an emergency setting to a target spec. NVIDIA trained for it, published the tensor-level breakdown, and named the machine it was designed to fit. The RTX Spark platform it targets has 128 GB of unified memory and 300 GB/s of bandwidth, and the model was written to run inside 24 GB of it.
For anyone building locally, the practical reading is short. Buy bandwidth if you intend to run models at 2 bits. Buy capacity only if you intend to run them at 4 bits or higher. And if the plan is to stream weights from system RAM, measure the channel first, because a copy test on this board returns 17.7 GB/s against a 42.7 GB/s spec sheet, and the model card does not mention the memory tier at all.
