On October 12, 2026, AMD published a rack specification at the Open Compute Project Global Summit. The document describes a 21-inch, 44-OpenU enclosure built to Open Rack V3. It is not a product AMD sells. OEMs build against it.
The part that changes the economics is not the chassis. It is a component inside the software stack: a model router that decides, per request, whether an agent’s call goes to a hosted frontier model or to an open model already running in the rack.
Two configurations are specified. A CPU-only rack on sixth-generation EPYC processors supporting up to 32,768 cores, aimed at operators who need dedicated agent-execution capacity and connect to inference infrastructure over the network. An all-in-one rack pairing EPYC CPU blades with AMD Instinct MI350P inference nodes, supporting up to 16 MI350P GPUs per rack. Both support air cooling and direct liquid cooling. The design layers on top of the standing OCP standards: DC-MHS for host processor modules, DC-SCM for platform management, HPRv2 power shelves, OCP NIC 3.0, and Caliptra as the silicon root of trust. Partner-built configurations from Dell, Cisco, HPE, Lenovo, QCT, Supermicro, Wiwynn and Hon Hai are planned from November 2026.
Why the router sits in the infrastructure layer
In a normal LLM deployment, one CPU supports four to eight GPUs. The CPU feeds the accelerator. In an agent deployment, the CPU does the work between model calls: routing tool calls, querying databases, triggering external APIs, enforcing security policy. AMD’s own engineering blog describes that ratio moving toward 1:1. That is why a router is drawn at the rack level rather than inside the application.
Intel argued the same case at the same summit with a phrase worth quoting: “Put every prompt in its place.” Nicolas Dubé’s keynote described an orchestration layer that scores each request on required depth of intelligence, context length, latency, data sensitivity and sovereignty, cost and energy. His rack numbers: partner-built Xeon designs on Open Rack v3, with select Xeon 6+ configurations scaling beyond 46,000 cores per rack at 133.45 kilowatts. The metric Intel proposed is not core count. It is how many concurrent agents a rack sustains at target latency.
Arm proposed a measurement method for the same question, developed with Fujitsu, Meta and OpenAI, based on replaying recorded agent execution sequences instead of benchmarking single operations. Analysis published by Signal65 put Arm AGI CPU at 1.99x modeled performance per rack against AMD EPYC 9755 across seven representative agentic workloads. FuriosaAI showed the split physically: an Arm AGI CPU handling orchestration and tool execution, RNGD accelerators handling prefill and decode, host memory holding the context. Intel and SambaNova have separately proposed routing prefill, decode and agent execution to different processor classes.
Separating the stages by hardware type is no longer a vendor position. It is the pattern the whole summit converged on. The decision models that make that routing call at application level, from the small open decision models released for local agents to the frontier decision APIs, are the same idea one layer up.
The term the router does not have
Every router described at the summit optimizes a per-token price. A per-token price is not the cost of a call. A call has a fixed part. The conversation is re-sent, the prompt is processed, and the request spends time touching nothing.
I measured that fixed part on my own hardware: a Ryzen 9 5900X, 64 GB DDR4, an RTX 3090 at 936 GB/s, llama.cpp serving a 125B MoE at IQ2_XS through an OpenAI-compatible endpoint. The workload was a 12-call agent loop with a growing conversation and a fixed system prompt, each call returning a one-line decision about whether to call a tool. Same shape every call, so the numbers compare call to call. llama.cpp is one of the three backends in the 2026 LLM backend comparison, and it is the one that reports prompt and generation timings separately, which is what makes this measurement possible at all.
| Call | Prompt tokens | Served from cache | Prefill | Decode | Wall clock |
|---|---|---|---|---|---|
| First call, cold | 86 | 0 | 541 ms | 440 ms | 1,003 ms |
| Median warm call | 254 | 221 | 251 ms | 320 ms | 614 ms |
| Twelfth call | 396 | 362 | 244 ms | 327 ms | 596 ms |
Across the 12 calls the loop spent 3,294 ms in prefill. The same prompts processed without cache reuse would have cost 18,132 ms. That is a 5.5x difference, and it is the entire reason cached-input pricing exists at the API layer. When a provider lists a lower rate for cached input, it is charging for a prefill it did not have to run.
The second number is the overhead outside inference, which is wall clock minus the engine’s own reported prompt and generation time. Median per warm call: 26 ms against 614 ms of wall clock. That median is small. The tail is not. Two of the twelve calls spent 257 ms and 1,591 ms outside inference, and across the whole loop 2,210 ms of 9,498 ms was non-inference time, 23.3%. Excluding those two calls the share drops to 5.6%. Both numbers are true, and the difference between them is the point: the fixed cost of a call is small on average and unbounded in practice, and a router that dispatches on token price has no term for it and no way to know which backend already holds the prefix.
What the same call costs in dollars
Prices below are published per-million-token rates read from a live pricing-comparison page in October 2026, applied to the measured call shape: 254 input tokens, 40 output tokens.
| Provider and model | Published rate per 1M tokens (in / out) | Cost per call |
|---|---|---|
| CoreWeave, gpt-oss-120b (cheapest verified) | $0.03 / $0.17 | $0.000014 |
| Heabsy, Qwen3.8 27B (cheapest verified) | $0.04 / $0.30 | $0.000022 |
| Baseten, gpt-oss-120b (mid-tier) | $0.10 / $0.50 | $0.000045 |
| Gemini 4 Argon, introductory price | $2 / $10 | $0.000908 |
The spread is about 65x, and every number is small enough that one call does not matter. The router’s savings are a volume number, not a per-call number. At 10,000 calls a day the cheapest lane costs about $0.14 and the frontier tier about $9.08. Both are rounding errors at the scale where a router is worth building. What the router actually buys is latency, and latency is where a fixed cost lives.
AMD’s own testing, described by Dan McNamara to Fierce Network, produced a 43% reduction in token cost and response times up to three times faster for workloads shifted to local inference. The cost side is structural: a call dispatched to a model already running in the rack carries zero marginal API cost. The latency claim is the part that connects to the measurement above. A cold prefill on a shared endpoint cost 541 ms for 86 tokens. A warm one cost 251 ms for 254 tokens. Routing to a backend that already holds the prefix is where a three-times latency claim comes from.
The routing decision and the cache state are decided by different systems. A cheap provider that publishes no cached-input rate charges full prefill. A local box that already holds the conversation charges nothing for it. A router that cannot see cache state will send a warm call to a cold endpoint and pay the difference in wall clock, which is the part the user actually experiences.
The two lanes at desk scale
You cannot buy a Helion rack for a home office. The shape is still available at desk scale, and it is the same shape: one machine for orchestration, one for inference, or one machine doing both with unified memory. The best hardware for running local LLMs breakdown covers the full budget-to-enterprise ladder, so this section only covers the split.
The orchestration lane is the one most home setups skip. A CPU-first box with a PCIe slot is the small version of the CPU-only rack: MINISFORUM MS-01, a 14-core i9-13900H with 64 GB DDR5, two 10G SFP+ ports and a PCIe x16 slot that is electrically x8, running around $1,559 at check time. It is not an inference machine. It holds tool calls, retrieval and the network path to whatever does the inference, and it is cheap enough that separating the two lanes is a real option rather than a theoretical one.
The inference lane is where memory capacity decides what you can run. The MINISFORUM MS-S1 Max carries a Ryzen AI Max+ 395 with 128 GB of LPDDR5-8000 unified memory for about $3,879, and the HP Z2 Mini G1a is the same architecture at 32 GB for about $2,000. The price gap between those two is the price of context, not of compute. Both are unified-memory machines, so the capacity you buy is the capacity you keep, and the wafer diversion behind the 3x DRAM price is why a 128 GB unified-memory box costs roughly twice a 32 GB one.
What to check before trusting any of these numbers
- The 43% figure is AMD internal testing. It has not been independently verified at rack level, and AMD published no agent benchmark data at the summit. NAND Research flagged that gap directly.
- Signal65’s 1.99x is a modeled rack figure built from server-level measurements, not a production result.
- Intel’s citation of open-weight models rising from roughly 31% to about 78% of observed token traffic between early July and late September comes from one gateway, Vercel AI Gateway. It is directional, not the market.
- November 2026 availability for a rack architecture of this complexity is aggressive, and buyers will purchase from OEMs building against a shared specification rather than from AMD.
The practical decision for anyone running agents locally is smaller than the keynote framing. Measure the fixed cost of your own calls before routing anything. If a call spends a quarter of its wall clock outside inference, a router that optimizes dollars will make the wrong trade on short calls, and it will make it invisibly, because the cost it saves is on an invoice and the cost it adds is in latency.
