What was announced
On November 3, 2026 Supermicro announced a collaboration with SambaNova to build and sell disaggregated inference systems, with an initial plan of 10,000 SN50 RDU chips. Supermicro integrates, tests, delivers and supports the systems, including on-site service. SambaNova engineers get models running and tuned on the SN50 and help with deployments. The first customers are NeoClouds and sovereign clouds, then enterprises running AI in their own data centers.
The architecture is the part worth reading carefully. In these systems NVIDIA GPUs handle prefill, the pass that processes the incoming prompt. RDUs handle decode, the pass that generates the answer one token at a time. That is not a positioning statement. It is a response to a measured property of agentic workloads: the two phases have different bottlenecks, so putting them on the same chip wastes one of them.
SambaNova’s own Hot Chips 2026 analysis put decode at 75 percent to 97 percent of model-inference time across six frontier-model configurations. At the high end, a DeepSeek-V3 case with an 8K input, 1K output, B300 FP4 and one active request spent 97 percent of inference time in decode. The company’s argument is that peak compute does not predict the speed a user feels. Installed memory bandwidth multiplied by the share of it that actually moves weights and KV cache is what does.
Where that claim holds, and where it does not
The 75-to-97 percent figure is a property of a request shape, not of agentic work in general. It holds when the prompt is short and the output is long. It stops holding the moment the prompt gets long, which is exactly what agents do: they accumulate context, call tools, and re-send a growing prompt every turn.
I measured the split on my own serving box instead of quoting the vendor. The engine is Strata, running a 125B MoE on a Ryzen 9 5900X with an RTX 3090 and 64 GB of DDR4. The requests were sent from a different machine, a Ryzen 7 2700X, so every wall-clock figure below includes the network round trip. Each row is the median of five passes with distinct content per pass, so the prompt cache was cold on every request (cache_n = 0 throughout).
| Request shape | Prompt tokens | Output tokens | Prefill ms | Decode ms | Decode share of wall | Prefill tok/s | Decode tok/s, engine | Decode tok/s, wall |
|---|---|---|---|---|---|---|---|---|
| Cold prefill, 1 output token | 5,262 | 1 | 2,470.8 | 16.2 | 0.6% | 2,130 | 61.7 | 0.4 |
| Short prompt, 512 output tokens | 373 | 479 | 819.4 | 4,031.7 | 82.3% | 455 | 115.7 | 93.3 |
| Agent-shaped call, 8,382 prompt tokens, 200 output | 8,382 | 200 | 4,048.8 | 1,790.0 | 30.0% | 2,070 | 111.7 | 33.6 |
Three shapes, three completely different answers. The same engine spends 97.4 percent of wall time in prefill on a long prompt with a one-token answer, 82.3 percent in decode on a short prompt with a long answer, and 30.0 percent in decode on a realistic agent call. The prefill-to-decode rate ratio on identical hardware was 34.5x, 3.9x and 18.5x across those three shapes.
Two numbers in that table must stay separate. The engine-reported decode rate is the rate inside the generation window. The wall-clock rate is tokens delivered divided by the whole request, including prefill. On the agent-shaped call those are 111.7 and 33.6 tokens per second. The gap is not request overhead. It is prefill sitting in the denominator, which is exactly what an agent loop experiences when it waits for a tool call to come back.
Speculative decoding is why the decode figures on this box are higher than a plain single-token loop would produce. Draft acceptance was 79.5 percent on the decode-heavy shape and 73.7 percent on the agent-shaped call. That is a per-request measurement on one engine on one machine, not a general property of MoE decoding.
What the SN50 actually is
The datasheet specs SambaNova published for the SN50 describe a chip built around dataflow rather than a sequence of separately launched kernels. TSMC 5nm, 432 MB of distributed on-chip SRAM, 64 GB of HBM2E, and DDR5 as a third memory tier. The compiler maps the model across a tiled mesh of compute and memory units. Double-buffered memory units move the next model tile while the current computation runs, and collective communication can terminate in SRAM instead of forcing another trip to HBM.
The network is where the “at scale” part lives. Inside an eight-socket node, seven integrated 800G Ethernet links form a fully connected topology, so each RDU has a direct path to every other RDU in the node. Two additional 800G ports per RDU connect to the inter-node scale-up fabric. Across all three domains the SN50 integrates ten 800G Ethernet ports with an aggregate 2 TB/s per RDU. A separate 400G RoCEv2 NIC handles scale-out between scale-up domains, and that layer is what carries KV-cache transfer for disaggregated inference.
SambaNova’s scaling example is the most concrete number in the whole announcement. Modeling DeepSeek-R1 from 64 to 256 RDUs lifts per-user speed from 200 to 500 tokens per second while model bandwidth utilization holds between 44 percent and 51 percent. Adding chips raises theoretical bandwidth. Per-user speed only rises if utilization survives it.
The February 2026 claims of 5x faster and 3x lower total cost of ownership against “competitive chips” are SambaNova’s own, and the comparator was never named. Treat those as marketing until someone reproduces them on a named card.
Why a single machine cannot do the split
Disaggregation works because two machines with different bottlenecks can be assigned to two phases. A local build is one memory system, so it cannot. What it can do is be bought for the shape it actually runs, and that is where the measurement above becomes a buying decision rather than a curiosity.
If your workload is short-prompt generation, decode is 82 percent of your wall time. Memory bandwidth is the constraint and the GPU’s compute sits mostly idle during that window. If your workload is long-context agent calls, prefill is 68 percent of your wall time and arithmetic matters again. The same box is the wrong purchase for both, and most people buy for the first shape while running the second. The tradeoff is the same one described for cards that skip HBM entirely: the memory tier you pay for has to match the phase that dominates your requests.
The third memory tier is the practical version of this. On a dataflow system the tiers are on-chip SRAM, HBM and DDR5. On a single machine the third tier is your system RAM, and the DDR5 price ladder is the whole cost of that tier. A Kingston FURY Beast 64 GB kit, 2×32 GB, 6000 MT/s, EXPO, desktop UDIMM, is around $1,199.99 on the price tracker as of October 3, 2026, 4.6 times the $260.76 price the same kit held when tracking began in July 2025. That is not a performance decision. It is the cost of the tier that holds the model while the GPU works on it.
The fourth tier is the SSD, and it is where the KV cache spills when the prompt gets long enough to matter. On the same engine, the shape that spent 68 percent of wall time in prefill was also the shape that produced the largest KV cache. Block size is what decides whether that tier behaves at 218 MB/s or 2,349 MB/s, so a queue-depth-aware drive is a different purchase than a large one. A TEAMGROUP MP44 2 TB M.2 2280 PCIe 4.0 drive is around $325.99 on the tracker as of October 21, 2026.
For the prefill side, compute is the cheap part of the trade. A renewed RTX 3090 with 24 GB of GDDR6X on a 384-bit interface is around $1,899.99 as of November 5, 2026. It is not fast by 2026 standards, but 2,130 tokens per second of prefill on a 125B MoE is not a bad result on a card that costs less than one RDU would, and the decode side on that same card still clears 100 tokens per second inside the generation window.
What the announcement does not say
Nothing in the press release states a price, a per-chip throughput figure, or a power figure per rack. It says RDUs serve large models without requiring extreme rack power density, which makes them easier to deploy in existing data centers. That is the actual commercial argument: a facility that already has 15 kW to 20 kW per rack can add decode capacity without rebuilding its power and cooling, and it can keep using GPUs it already owns for prefill.
For anyone running models locally, the transferable part is not the chip. It is the measurement. Run a long prompt with a one-token answer and a short prompt with a long answer against your own endpoint, and compare the two. If the second one is where your time goes, you have a bandwidth problem and a faster card will not fix it. If the first one is where your time goes, you have a compute problem, and the memory upgrade is the wrong spend. The 75-to-97 percent decode share is real for the shape SambaNova measured. Your shape is yours, and it takes two requests to find out which one you are actually running.


Pingback: 77.1 Tokens per Second on a Rented Card: What a Measured Single Stream Says About the Break-Even Line - GenX AI Tools
Pingback: 21,333 Tokens per Second Is a Batch Number: What a 394.7 TB/s Requirement Says About the Single Stream - GenX AI Tools