Renting an Nvidia B200 costs more this month than it did in July. Two trackers agree on the direction and disagree on the size. SemiAnalysis recorded a 79 percent rise in on-demand B200 rates, reaching $8.01 per GPU-hour by the end of September 2026. The Ornn Compute Price Index covers a slightly later window, late July to late October, and shows the same card moving from $6.58 to $9.76 per GPU-hour, a 48 percent increase. Live listings in early November put the spread at $1.89 to $16.11 per GPU-hour across 10 to 20 providers, the top held by Google Cloud and AWS above $14, with typical on-demand medians between $6 and $8.50.
Most coverage of this price move stops at the hourly rate. The hourly rate does not decide a bill. Tokens per hour does, and for most readers that is a single stream on one card.
The formula everyone uses, and what it leaves out
The break-even calculation is simple enough to fit on one line:
Effective cost per 1M tokens = (GPU price per hour ÷ sustained tokens per hour) × 1,000,000
Qovery worked it through with real inputs in November 2026. gpt-oss-120B on eight B200 cards produced 90,205 tokens per second in the MLPerf Inference v6.1 Server scenario, a CoreWeave submission, which is 324.7 million tokens per hour at full utilization. Against Together AI’s published $0.60 per million output tokens, self-hosting wins above roughly 33 percent sustained utilization on Google Cloud DWS Flex-start and roughly 58 percent on AWS on-demand. Below that line the API is cheaper.
Every variable in that calculation is sourced except one. “Sustained tokens per hour” is a benchmark number measured with the machine kept busy. It is not what a single user, or a single agent loop, produces. That is the gap this article is about, and it is the gap that turns a favorable-looking hourly rate into a worse per-token cost than the API.
What I measured on my own hardware
I ran a real request against the inference engine I run on. The serving box is a Ryzen 9 5900X, 12 cores, with an RTX 3090 holding 24,576 MB of VRAM. The model is Strata-Flash-Next, a 125B MoE that lives in about 12 GB of VRAM and the rest across system RAM, CPU and SSD. The requests came from a different machine, a Ryzen 7 2700X, over the local network, so every figure below includes the network round trip. The API returned no timing block, so these are wall-clock numbers through the API, not engine-reported numbers. That distinction matters: wall-clock through the API is what an agent loop experiences.
Two loops, because they measure different things. The latency loop sends one request at a time with no overlap. The throughput loop overlaps submission across concurrent requests. The same card produced different per-request costs under each. Five passes per level, median.
| Concurrent streams | Throughput loop (tok/s) | Latency loop per-request wall (s) | Single-stream decode (tok/s) |
|---|---|---|---|
| 1 | 77.1 | 1.990 | 105.9 |
| 4 | 115.7 | — | — |
| 8 | 117.7 | — | — |
The single-stream decode figure isolates the generation window by subtracting the measured prefill from the wall time. The prefill subtracted is the warm one, 0.102 seconds, because the prompt was repeated across passes. It is higher than the throughput loop’s single-stream figure, which counts the whole request window including submission and tokenization. Both are real measurements. They are not the same measurement, and quoting one as the other is how these tables end up wrong.
The row that matters is the first one. A single agent loop on this machine runs at 77.1 tokens per second under the throughput loop, which is 65.5 percent of the eight-stream rate the same hardware reaches when kept busy. A rented card running one stream is not at 100 percent utilization. It is at roughly two thirds of its own ceiling, and the per-token cost scales by the reciprocal of that share.
What that costs per million tokens
Rental rates for the same card, read in November 2026. GPU Economy’s median rate card for an RTX 3090 is $0.45 per GPU-hour across 11 providers. OpenRelay lists $0.18, the lowest on-demand rate of those 11. GrabGPU’s marketplace median across 6 providers is $0.16 with a floor of $0.15. Salad lists $0.09 at lowest priority. Two trackers, the same card, 1.8 times apart on the same day.
| Setup | Rate read | Cost per 1M tokens |
|---|---|---|
| RTX 3090, one stream end to end, market median | $0.45/GPU-h | $1.62 |
| RTX 3090, one stream end to end, marketplace median | $0.16/GPU-h | $0.576 |
| RTX 3090, one stream end to end, lowest listed | $0.09/GPU-h | $0.324 |
| RTX 3090, generation window only, market median | $0.45/GPU-h | $1.18 |
| B200 at MLPerf throughput, SemiAnalysis rate | $8.01/GPU-h | $0.197 |
| B200 at MLPerf throughput, Ornn late-October rate | $9.76/GPU-h | $0.240 |
| Per-token API (Together AI, gpt-oss-120B output) | $0.60 | $0.60 |
The B200 row is 11,276 tokens per second per GPU, which is 90,205 divided by eight. That row is not comparable to my machine as a rate. The B200 holds a model fully inside 180 GB of HBM3e; my card runs a 125B model that does not fit in 24 GB and is served across four memory tiers. A model that fits in VRAM on the same 3090 decodes faster than my split-tier path. The $1.18 figure is a cost for my workload, not a general rate for that card.
What the comparison does show is the size of the gap. The B200 hour is 17.8 times the 3090 median hour. Against my generation-window rate of 105.9 tokens per second the throughput gap is 106 times, and a price gap of 17.8 against a throughput gap of 106 means the expensive card is cheaper per token by a factor of 5.98. Against the end-to-end rate of 77.1 the gap is 146 times and the factor is 8.2. That argument survives when the cheap card is kept busy.
It does not survive when the cheap card runs one stream. At the market median, my single stream costs $1.62 per million tokens end to end against an API that charges $0.60. The break-even rental rate for my workload is $0.17 per GPU-hour on the end-to-end rate, or $0.23 if you count only the generation window. Below that line renting beats the API, above it the API wins. The market median is $0.45 and the marketplace median is $0.16, which sits just under the line. The decision flips on which rate you can actually get, not on which card is faster.
The utilization line nobody controls
Qovery’s break-even assumes you can keep the hardware busy. A GPU node used 40 hours a week but left running for all 168 hours spends 76.2 percent of its time producing nothing. At the $0.45 median that turns my $1.62 per million into $6.81 per million, a factor of 4.2.
Per-token pricing bills nothing while a GPU sits idle. That is the structural difference between the two models and why the hourly rate is the wrong thing to compare. A per-token API price contains the provider’s utilization assumption. A rented GPU-hour contains yours.
Prefill is the part an agent loop pays per call
Decode rate gets the attention because it is the visible number. For agent work, first-token cost is per call, and an agent makes many calls against the same context.
| Prompt condition | Median wall to first token | Effective prefill rate |
|---|---|---|
| Cold, every prompt unique | 1.531 s | 1,067.9 tok/s |
| Repeated identical prompt | 0.102 s | 16,029 tok/s |
The second row is not a prefill rate. It is a prefix cache hit: the engine answered the same 1,635 tokens in a tenth of a second because it had already processed them. The first row is the cost of a first call.
Over a 50-call agent session against one context that is 76.6 seconds cold against 5.1 seconds warm, a factor of 15.0. A provider that bills cached input at a discount is pricing exactly this. Cached input is the cheapest token in the industry because the work was already done.
Memory is why the rates moved
The rental increase is not a mystery. Counterpoint’s Q3 2026 memory tracker put global DRAM revenue at nearly $200 billion, up 369 percent year over year, with bit shipments up about 4 percent quarter over quarter. Average selling price reached $1.9 per gigabit. Gartner reports memory prices more than doubled over 12 months. A market that grew fivefold on price while shipping the same number of bits is a supply allocation story.
HBM3E 36GB 12-Hi stacks were quoted at $2,100 per stack on November 9, up 14 percent year over year. Samsung said next year’s HBM supply is sold out. SK hynix said client requests exceed twice its available capacity. When a card’s memory is contracted two years ahead at a floor price, the hourly rate for the card holding it moves.
This is the same mechanism as the cache cliff on a smaller machine. A working set that stays inside the 64 MB of L3 on my serving box never touches DRAM, and the rate measured inside cache cannot be expressed as a share of the memory channel. Once it crosses the cache, the rate falls to what the channel can actually deliver. The rental market is the same physics priced at the tier you paid for.
The same pressure reaches the tier a home build buys. A 64 GB desktop UDIMM kit, G.SKILL Trident Z RGB 2×32 GB DDR4-3200 CL16, is $279.99 as of November 11, 2026, which is $4.37 per gigabyte. That is the tier a split-tier engine spills into when the model does not fit in VRAM. It is priced per gigabyte while the rental market prices per throughput.
Where a buyer actually stands
The purchase price and the rental rate disagree about the same two cards. An RTX 5090 with 32 GB costs $7,499.99 on a live listing as of November 11, 2026. A bandwidth per dollar comparison against a 3090 at $2,162.99 is a factor of 3.47. The rental market prices the same pair 1.80 times apart. That disagreement is a factor of 1.93, and it exists because renting is priced on throughput rather than on capital.
Per gigabyte of memory-hour the numbers are cleaner. A 3090 hour at $0.45 is $0.01875 per GB-hour. A B200 hour at $8.01 is $0.04450 per GB-hour, 2.37 times higher. A 5090 hour at the $0.81 median across 24 providers is $0.02531 per GB-hour, 1.35 times the 3090. A workload bounded by memory capacity benefits from the cheaper card. A workload bounded by throughput does not.
For a single stream on hardware you already own: your effective rate is what one stream produces, not what the machine produces when kept busy, and that is roughly two thirds of the ceiling. Tokens per watt is the other half of the per-token cost no rental table shows, because a rented hour includes the power and a purchased card only shows it on a bill.
Three things decide it, in order. Whether the model fits in the memory you paid for, because a model that spills to the next tier changes the rate before the price does. Whether you can keep the card busy, because idle hours multiply the rate by 4.2 at a 40-hour week, which is the difference between $1.62 and $6.81 per million on the same hardware. And whether your workload is one stream or many, because a single agent loop is one stream. The hardware ladder is the wrong starting point for that decision. The request shape is. A decode-bound single stream on a 24 GB card is a different problem from a decode share measured across a rack.

