Maia 300: What an 18 TB/s Inference Chip Says About the Decode Bottleneck

What was announced

On October 12 Microsoft detailed Maia 300, its third-generation inference accelerator, and attached the announcement to a multi-gigawatt rollout starting in 2027. The chip is not a general-purpose GPU. It is built for one job: answering prompts. Microsoft states plainly that Maia 300 will become the primary AI inference system for its own products and services.

The published specifications are specific enough to argue from. The compute die is built on TSMC N2P, a 2nm-class node, with memory-controller and input-output functions on separate dies. Four DDR RDIMM tiles serve as system-level main memory. Eight HBM4 stacks provide 288 GB and 18 TB/s at a 9 Gbps pin speed. The chip delivers 6.9 petaflops of FP8 and 13.8 petaflops of FP4 at a 950 W maximum TDP. Ninety-six accelerators fill a 167 kW rack, and optical switches join up to 128 racks into a superpod of 12,288 accelerators.

Production deployments begin in 2027. Nothing here is available to buy.

Which design choices actually address the decode bottleneck

A language model answers in two phases. Prefill processes the prompt and is compute-heavy. Decode generates the reply one token at a time and spends most of its time waiting on data fetched from memory. Longer contexts enlarge the state that must be stored, and agents trigger many calls in succession. That is why the design effort moved to the second phase.

Almost every architectural choice targets that second phase, and the ratio of the claimed gains makes the intent obvious: compute FLOPs up 36 percent, memory bandwidth up 250 percent. The design is not chasing arithmetic. It is chasing bytes moved per token.

The mechanisms that do real work:

• Non-uniform memory across chiplets. Maia 200 used a uniform organization. Maia 300 gives each compute region nearby memory so software places data next to the compute that uses it. Shorter travel distance means less energy per move, which is the lever on a memory-bound workload.
• Bandwidth over capacity. Eight stacks at 18 TB/s is 2.5 times Maia 200 and roughly twice the bandwidth per watt. Capacity grew 33 percent while bandwidth grew 2.5 times.
• Narrow precision with hardware conversion. Enhanced MXFP4 and sub-3-bit formats reduce the bytes moved per token, and the conversion happens in hardware rather than software. Fewer bytes per token is more tokens per second on a fixed memory subsystem.
• Instruction support for the kernels decode runs. The vector engine adds optimizations for softmax and low-precision casting. These are decode kernels, not training kernels.
• Prefill and decode sized independently. Workers are placed separately and key-value state moves over the scale-up fabric instead of leaving through a host-mediated path. A long-document service gets more hardware for prefill, a chat service more for decode. That is scheduling, not silicon, and it is where much of the claimed gain lives.
• Commodity DRAM as the cold tier. The four RDIMM tiles are the quiet tell: even a hyperscaler inference chip does not put everything in HBM.

Which claims are still marketing

Three numbers carry the announcement: up to seven times the rack-scale token throughput of Maia 200, more than 2.5 times the performance per watt, and 40 percent more tokens per megawatt than leading GPU-based systems. All three are Microsoft’s own figures.

The seven-times figure is rack-level: the rack holds twice the accelerators, so the per-chip gain lands near 3.5 times, consistent with the memory numbers. The 40 percent figure is weaker. Microsoft’s own deep dive names Maia 200 as its only comparator and publishes no test conditions for the GPU claim. No matched models, precision, context lengths, no independent reproduction, no measured dollars per token.

Microsoft names no outside customers for Maia and does not quantify what share of its traffic the chip will carry. It remains committed to Nvidia-powered capacity through IREN, a five-year contract worth about 9.7 billion dollars for GB300 systems supporting 200 MW of load. The first evidence of whether the efficiency claims hold will come from someone else’s benchmark.

The same physics on a desktop, measured

Decode being memory-bound is not a hyperscaler problem. It is the same constraint on a home machine, and it is measurable on one request. I ran a real prompt against the llama.cpp endpoint on the operator’s box: a Ryzen 9 5900X, 64 GB of DDR4, and an RTX 3090 serving a 125B mixture-of-experts model that runs across GPU, system RAM, CPU, and SSD. The prompt was a fetched 3,183-character industry article, not a greeting, so it measures prefill rather than request overhead.

Results:

• 806 prompt tokens, 300 completion tokens.
• Engine-reported prefill: 647.3 tokens per second, taking 1.25 seconds.
• Engine-reported decode: 92.0 tokens per second, taking 3.26 seconds.
• Wall-clock decode from the client, first token to last: 91.3 tokens per second. Total request wall time 4.531 seconds, of which 26 milliseconds was anything other than inference.
• Speculative decoding: 234 draft tokens generated, 160 accepted, 68.4 percent acceptance. The throughput figure above counts generated tokens including rejected drafts, so the delivered-token rate is lower.

Two follow-ups matter as much as the headline numbers. Sending the identical prompt again hit the prompt cache: 801 tokens served from cache, prefill dropped from 1.25 seconds to 41 milliseconds, request time from 4.531 to 2.991 seconds. A cache hit is the cheapest bandwidth on any system, and it is the problem Maia’s disaggregated prefill placement addresses at rack scale.

A short agent-style call shows what a loop pays. Five small calls with a 56-token prompt took 152 to 516 milliseconds each, with 16 to 22 milliseconds per call on request handling and tokenization rather than inference, about 10 percent of a short request. On a long request that overhead is noise. On an agent loop it is the cost.

The arithmetic that decides it

Dividing the RTX 3090’s published 936 GB/s of memory bandwidth by the measured 92 tokens per second gives about 10 GB of data touched per generated token. That is more than the quantized active weights alone explain, and the gap is where the cost sits: key-value state read per token, draft work, and layers offloaded because 24 GB is not enough for a 125B model. The memory subsystem sets the ceiling. The FLOPs do not.

Maia 300 buys that ceiling directly and pays for it at HBM prices. Industry estimates cited by Seoul Economic Daily put a 12-high 36 GB HBM4 stack at around 1,300 dollars next year against about 600 dollars this year, or 36.11 dollars per gigabyte on a contract basis, 116.6 percent above the January 2026 level. At those estimates the memory on one Maia accelerator is worth about 10,400 dollars. Micron said on its fiscal fourth-quarter call that it has committed more than 75 percent of FY2027 volume at prices well above 2026 levels.

Conventional DRAM is not exempt. TrendForce projects contract prices up 10 to 15 percent quarter over quarter for the fourth quarter of 2026 and NAND up 15 to 20 percent, with server DRAM still undersupplied. A memory price index set at 100 in the third quarter of 2025 sits above 520 a year later. That is the mechanism behind what a 3x DRAM price means for local LLM builds: HBM competes for the same wafers as desktop memory, and the price signal reaches the retail kit first.

Power is the other half of the envelope. A Maia rack draws up to 167 kW for 96 accelerators, about 1.7 kW per accelerator once the rest of the rack is counted. The same week Nvidia, AMD, Google and Meta published the Open Data Center Specification v0.7 covering power, cooling and layout, plus the OCP Grid Resilience Specification, which requires large AI sites to return to near-full load within two seconds of a grid disturbance. Those standards exist because power, not silicon, is the binding constraint on deployed inference. On a single card at home the question is smaller but identical, and it is measured in what power capping actually costs a local LLM build.

Where the bandwidth ladder actually sits

Capacity is the cheap axis and bandwidth is the expensive one. Bandwidth is fixed by the memory subsystem you already bought, and an engine that splits experts across GPU, RAM, CPU and SSD pays for that on every token, because a lookup table read per token is part of the inference path. The buying tiers are in best hardware for running local LLMs in 2026 rather than restated here.

What to do with this

For anyone choosing hardware for local inference, Maia 300 is not a purchase decision. It is a signal about which axis the industry is spending on, and that axis is the same one you are on.

First, buy bandwidth before capacity. A card fast enough to hold the model you intend to serve beats a bigger pool read through a slower path. The EVGA RTX 3090 24 GB GDDR6X at about 1,850 dollars is the cheapest way to get 936 GB/s on a desktop, and it is the card these numbers were measured on.

Second, if the model does not fit in the card, unified-memory machines are the capacity answer rather than the bandwidth answer. The GMKtec EVO-X2 with 128 GB LPDDR5X-8000 runs at about 3,500 dollars. It holds a model a 24 GB card cannot and serves it more slowly per token, the same trade Microsoft made with HBM for hot data and RDIMM for cold.

Third, system memory is now priced as a scarce good, and the kit price reflects it. A Crucial Pro 64 GB DDR5 6000 desktop UDIMM kit is about 1,120 dollars today, which is a lot for 64 GB of unbuffered memory. If you are building for offload, that cost is the price of the tier your engine reads per token. Storage is the one part of the path that has not repriced as aggressively: a WD_BLACK SN850X 2 TB with heatsink

Maia 300 is a well-designed chip for a genuinely memory-bound workload, and its engineering claims are coherent. Its efficiency claims are unverified. For anyone building locally, the interesting part is not the seven-times number. It is that the most expensive component on the package is memory, and that the same constraint shows up on a desktop as a 10 GB-per-token bill paid to whichever memory subsystem you chose.