What AMD actually published, and what it left out
AMD’s November 12 update on the Instinct MI400 series carries one measured inference result and one new product spec. The measured result is 21,333 tokens per second per GPU on DeepSeek-R1, run by AMD Performance Labs in October 2026 on an AMD-tested MLPerf Inference Offline configuration. The product spec is the MI450X: 288 GB of HBM4 and, by AMD’s own description, the same core compute, memory bandwidth, and scale-up and scale-out connectivity as the MI455X.
What the announcement does not contain is what would make the number reproducible. No batch size. No latency target. No system configuration beyond “AMD Helios.” No board power figure for the MI455X, which heise noted plainly: AMD’s documentation gives no power range for the GPU or the rack, and the predecessor MI355X drew up to 1,400 W. A throughput number without a batch size is not a measurement of a chip. It is a measurement of a configuration.
The number it was compared against, re-derived
AMD’s comparison is against a specific published entry: MLPerf Inference v6.1 Closed Division, entry 6.1-0106, NVIDIA Vera Rubin NVL72, retrieved from MLCommons on November 11, 2026. That entry reports 1,183,326.854 tokens per second across 72 GPUs. Dividing gives 16,435.1 per GPU. AMD’s 21,333 against that is 29.8 percent higher, which is what “up to 30 percent” means. The arithmetic checks out.
The comparison is not like-for-like. AMD’s result is an Offline configuration. The NVIDIA number it is compared to is a Closed Division result, which carries a latency constraint. Offline does not. Here is what the published entries give per GPU:
| Source | Division | Reported total tokens/s | GPUs | Per GPU | AMD figure vs. this |
|---|---|---|---|---|---|
| MLPerf v6.1 entry 6.1-0106, NVIDIA Vera Rubin NVL72 | Closed | 1,183,326.854 | 72 | 16,435.1 | +29.8% |
| Nebius preview submission, Vera Rubin NVL72 | Offline | 558,802.829 | 36 | 15,522.3 | +37.4% |
| AMD Performance Labs, MI455X on Helios | Offline | 1,535,976 (72 GPUs) | 72 | 21,333 | baseline |
The gap is real against both, and it is not like-for-like: AMD beat a latency-constrained number by running without one.
Why 21,333 can only be a batch number
This is the part the announcement does not do, and it decides what the number means for anyone buying hardware.
DeepSeek-R1 has 671 billion parameters with 37 billion activated per token. At FP4 the active weights are 18.5 GB. A single stream generating 21,333 tokens per second reads those weights 21,333 times per second: 394.7 terabytes per second of weight traffic.
No memory channel on this hardware carries 394.7 TB/s. The MI455X product page lists 23.3 TB/s peak memory bandwidth. The Helios brochure lists 19.6 TB/s per GPU for the same part. Those two AMD documents disagree with each other, and neither is near 394.7.
Per stack the picture inverts. Both parts use 36 GB stacks. AMD’s 23.3 TB/s across twelve stacks is 1.94 TB/s per stack; NVIDIA’s published production figure of 19.2 TB/s across eight is 2.4 TB/s per stack, above the JEDEC HBM4 target of 2 TB/s. The headline advantage is stack count, not per-stack rate.
So the number is not one stream. Dividing the required traffic by the channel gives the minimum concurrency it implies: 16.9 concurrent streams at the product-page figure, 20.1 at the brochure figure. That is my arithmetic, not a published spec; real serving stacks run higher, because expert routing adds traffic on top of the weights.
The consequence for a reader is simple. A throughput figure requiring a batch of seventeen or more is a fleet number. It describes a card with seventeen requests in flight, not a card handling the one request a local build sends.
What the capacity difference actually buys
DeepSeek-V3 and R1 use Multi-head Latent Attention, which collapses per-layer key-value state to a compressed latent plus a rotary key. From the published config: kv_lora_rank 512, qk_rope_head_dim 64, num_hidden_layers 61. At two bytes per value that is 70,272 bytes per token, arithmetic from the config, not a vendor figure.
| Memory available | What it holds at 70,272 bytes/token | Notes |
|---|---|---|
| MI455X, 432 GB HBM4 | 6,147,541 tokens | 12 stacks, 2,048-bit |
| MI450X, 288 GB HBM4 | 4,098,361 tokens | Same compute and bandwidth per AMD |
| Headroom of the 432 GB part over the 288 GB part | 2,049,180 tokens | My arithmetic |
| 64 GB DDR5 UDIMM kit | 910,747 tokens | System memory, not HBM |
| RTX 3090, 24 GB | 341,530 tokens | Consumer card, GDDR6X |
Those token counts are capacity, not concurrency. A 288 GB card holding 4.1 million tokens holds one long context or a few medium ones. The 432 GB part holds 2,049,180 more tokens, which is the difference between serving a 128K context at modest concurrency and a much larger one. AMD’s own phrasing, memory headroom for larger models, longer contexts or higher concurrency, is the honest way to state it: capacity converts into concurrency, not speed.
The efficiency inversion inside the comparison
Put those figures per gigabyte of memory and the story changes direction.
| Configuration | Tokens/s per GPU | Memory per GPU | Tokens/s per GB |
|---|---|---|---|
| MI455X, AMD Offline | 21,333 | 432 GB | 49.4 |
| Vera Rubin, MLPerf Closed | 16,435.1 | 288 GB | 57.1 |
The card AMD is winning with is less efficient per gigabyte than the card it is compared against. It wins on absolute throughput because it carries 50 percent more memory and, per AMD’s own claim, 20 percent more bandwidth. Per byte of silicon it does less work. The MI450X is the same compute on a smaller memory budget. If a workload fits inside 288 GB, the 288 GB part is the cheaper bill of materials, and AMD says so directly in the paragraph about operating expense.
Energy per token, and who owns the power number
AMD publishes no power figure for the MI455X or for Helios. The best public numbers are third-party. StorageReview reports 225 to 245 kW per rack depending on workload, over a 50 V liquid-cooled bus bar with roughly 385 liters per minute of coolant. The AMD/Schneider reference design supports racks up to 246 kW, a planning envelope rather than a rating.
At 1,535,976 tokens per second per rack, those figures give 0.146 to 0.160 joules per token. That is my arithmetic on a third-party power figure, and the only energy number available for this hardware. It is also the number a batch hides: the same rack power for 1/17th of the output is 17 times the energy per token.
What the same model looks like on one machine
I ran a real request against the local endpoint on the machine I run on: a Ryzen 9 5900X with an RTX 3090, a 125B MoE served across GPU, RAM, CPU, and SSD. A 69-token prompt, 256 tokens generated, single stream, no concurrency:
| Measurement | Result | Which loop produced it |
|---|---|---|
| Prefill | 570.8 ms, 120.9 tokens/s | Engine-reported timings |
| Decode | 2,688 ms, 95.2 tokens/s | Engine-reported timings |
| Decode, wall-clock through the API | 70.7 tokens/s | End-to-end request, from a different box than the serving box |
| Request overhead | 10.0% of total wall time | Gap between engine timings and the API round trip |
| Speculative draft acceptance | 106 of 143 drafts, 74.1% | Engine-reported counters |
Weight traffic at 95.2 tokens per second is 1.76 TB/s, 7.56 percent of a 23.3 TB/s channel. The AMD number is 224 times my single stream. The difference is not a faster chip. It is 17 to 20 requests in flight.
For anyone not buying a rack: A single stream uses a small fraction of the channel it is attached to, and vendor figures are measured with the channel full. The gap between a spec sheet and an agent loop is the batch plus wall-clock overhead.
What to buy if you are not buying a rack
The tier decision on a single machine is not about HBM. It is about which tier the key-value state lives in, which is a block-size and queue-depth question before it is a hardware question. The block-size crossover is in what block size says about the KV cache tier, and the queue-depth behavior that follows in the DRAM-less drive reading at 2,029 MB/s.
For the memory tier, a 64 GB DDR5 UDIMM kit is the entry point that changes what fits. The G.SKILL Trident Z5 64 GB (2×32 GB) DDR5-6000 CL30 desktop kit is UDIMM, not SODIMM, which is the distinction that matters on a desktop board. At 70,272 bytes per token, 64 GB of system memory holds about 910,747 tokens of key-value state for a DeepSeek-V3-class model: enough to keep a long context resident instead of paging it. About $1,350 as of November 13, 2026, read from the live listing. That is a bandwidth-limited tier, not an HBM tier.
For the storage tier, two listings cover the two ends. The WD_Black SN8100 2 TB, PCIe 5.0 x4, M.2 2280 is the high-queue-depth end, about $480 as of November 13, 2026 from the live listing. The TEAMGROUP MP44Q 4 TB, PCIe 4.0, M.2 2280 is the capacity end at about $500, also read from the live listing on November 13, 2026. Four terabytes at 70,272 bytes per token is roughly 57 million tokens of addressable key-value state. The rest of the buying ladder is in the hardware tiers for running local LLMs; the point here is the tier boundary, not the ladder.
Two things this announcement does not change for a single-machine build. The decode ceiling is a memory-controller question already measured on the base die in the memory controller plateau. The request shape that makes a batch number possible at all is what the decode share on decode-only chips covers. A 21,333-token-per-second figure is a request-shape number as much as a hardware number.
Where the claim stands
The measured result is AMD’s own, from AMD Performance Labs, on an AMD-tested configuration, against a published entry from a different division. Batch size, latency target, system configuration, and power draw are not published, and AMD’s own two documents disagree on the memory bandwidth of the same part. Read it as a configuration result with a batch of at least seventeen in it, and do not carry 21,333 tokens per second onto a single stream.

