A single memory chip package on a plain dark background

A 2 GB Board That Reserves 605 MB Before Generating One Token: What the Coralboard SL Says About Static Shapes

What actually shipped

Synaptics put the Coralboard SL into general availability on October 20, 2026 through DigiKey. The single-unit price is $99.79, with 1,913 units listed in stock on DigiKey at the time of writing. That is the first board to carry an implementation of Google’s open-source Coral NPU, and it is the cheapest entry point in the Coral line since the original dev board. The board is an Astra SL2619: two Cortex-A55 cores at 2 GHz, a Cortex-M52, a 1 TOPS Torq NPU, 2 GB of DDR4, a MicroSD slot, NOR flash, and camera and display connectors over CSI and DSI.

Price is not the interesting part. The interesting part is what a 270M transformer does to 2 GB of memory once the compiler has to commit to a shape before it can run anything.

What the datasheet says about memory

The Coral NPU has two on-chip memories: 8 KB of ITCM for instructions and 32 KB of DTCM for data. Both are single-cycle-latency SRAM, and the datasheet is explicit that this is the point. The design reduces energy by shortening the distance data travels inside the memory hierarchy rather than by moving more data per second. The compute side is a quantized outer-product multiply-accumulate engine, 4 x 8-bit multiplies reduced into 32-bit accumulators, 256 MACs per cycle, on a 32-bit RISC-V core with the Zve32x vector extension. BFloat16 is supported natively for transformer work.

32 KB is the whole working set. Nothing about a language model fits in 32 KB. Weights live in external memory, and the Synaptics product-line datasheet lists that interface as LPDDR4/4x, up to 2 GB, single rank, single channel. Synaptics publishes no bandwidth figure for the SL2619, so any GB/s number for this board is arithmetic, not a spec.

The arithmetic of a 270M model on 2 GB

Gemma 3 270M is 18 layers, 4 query heads, one key-value head, head_dim 256, hidden size 640, intermediate size 2048, a sliding window of 512 with full attention every sixth layer, and a 32,768-token context. The single KV head is what makes the cache small in absolute terms.

Per token, per layer, the cache grows by 2 (K and V) x 1 head x 256 dimensions x 2 bytes for bf16 = 1,024 bytes. Across 18 layers that is 18,432 bytes per token. At a full 32,768-token context that is 605 MB of cache, which is 576 MiB, or 28 percent of the board’s memory. Add the weights, roughly 540 MB in bf16, and the model is already past 2 GB before a single token is generated.

This is not a runtime problem. It is a compile-time commitment. Torq compiles decode with static shapes, which is the standard and efficient choice on an NPU, so the cache has to be a fixed tensor of shape [batch, heads, MAXSEQ, head_dim]. MAXSEQ is baked into the compiled model. The board reserves the whole buffer up front, whether the conversation uses it or not.

Why the reservation is also a per-step cost

A static-shape attention kernel computes over the whole MAXSEQ-length cache every step, causal-masked. So the reservation costs memory and time at the same rate. The developer working through this in the Torq issue tracker measured it directly: on a dense 350M model, dropping MAXSEQ from 256 to 64 made the decode body 2.4 times faster. That is the per-step cost scaling with the committed shape, confirmed by experiment rather than by theory.

The same thread is honest about what int8 does and does not buy. Int8 doubles the vector width of the matmul path in Torq, which is a real 2x ALU gain, and it halves the weight footprint, which is the reliable win. But the measured net decode speed on the dense model was lower with int8 than with bf16, because the per-matmul quantize-dequantize rescale and cast overhead exceeded the matmul gain. And the int8 path is currently blocked by a compiler bug at the int8-to-bf16 boundary: the same graph is correct on the host interpreter at cosine 0.99 and wrong on the board at 0.82. The developer switched to a Mamba-2 hybrid specifically to escape MAXSEQ, because a recurrent state is constant-size regardless of sequence length. That build works end to end at about 0.95 tokens per second.

What I measured on my own hardware

The usual claim is that decode throughput falls as context grows. I ran that against the engine I actually run on: a 125B MoE served on a Ryzen 9 5900X with an RTX 3090. Requests were sent from a different machine, a Ryzen 7 2700X, so the wall-clock column includes a network round trip and is not comparable to the engine-reported column. Same prompt content, same 64-token output, temperature 0, engine timings read from the streaming response.

Context tokens Engine prefill t/s Engine decode t/s Draft accepted Wall clock
59 107.8 78.5 29 / 45 1.16 s
2,071 1,482 88.1 35 / 53 2.17 s
10,071 2,140 98.0 34 / 53 5.44 s
20,071 2,386 106.6 34 / 48 9.18 s
40,071 2,451 99.2 33 / 50 10.57 s

Decode throughput did not fall with context. It rose, then settled. The last row is not a clean number because the engine reused 16,384 tokens of prompt cache from the previous run, so its prefill figure is inflated by cache reuse. The decode column is still valid.

The reason is the ratio. For a 270M dense model, the cache traffic per generated token is 18,432 bytes. The weights read per generated token are about 540 MB. Cache traffic is 0.003 percent of weight traffic at any context length. A model that small is not limited by cache growth. It is limited by streaming the whole model past the memory controller once per token. That is why the small-board numbers in the 273 GB/s bus measurements describe the same ceiling on local decode as this board: the bus is the constraint, and the cache is a rounding error on top of it.

The MoE I measured behaves differently for a different reason. It reads a small slice of active experts per token rather than the whole model, so its per-token weight traffic is far lower than a dense model’s, and speculative decoding carries it. That is not a general result about long context. It is a result about this engine and this model.

Where the board actually fails

Take the single-channel assumption at 3200 MT/s, which gives 6.4 GB/s. Streaming 540 MB of bf16 weights per token puts the ceiling at about 11.9 tokens per second. The project reports 1.95 to 2.1 tokens per second on the NPU path at MAXSEQ 256, which is about 17 percent of that ceiling. Its Q5_K_M build is a 248 MB file, giving a ceiling near 25.8 tokens per second, and it reports 9.7, which is about 38 percent of that.

So the board is not hitting its memory wall at this model size. On the NPU path the binding constraint is the software: the int8 graph that would halve the footprint does not execute correctly on the board, so decode runs as bf16 with the ALU penalty the compiler imposes on bf16 matmul. On the CPU path the binding constraint is two small cores doing arithmetic, not the bus. The memory wall becomes the binding constraint at the point where the model no longer fits in 2 GB at all, which is the failure this board hits first.

Every number in that paragraph is the project’s own measurement, not an independent reproduction. Treat it as reported until someone else repeats it.

What this means for a build decision

On a 2 GB board, the cache tier is the first thing that stops a build from loading, and the shape you commit to at compile time is the thing that decides whether it loads. A MAXSEQ of 256 costs 4.7 MB. A MAXSEQ of 32,768 costs 605 MB. The same model, the same weights, a 128x difference in committed memory, and a measured 2.4x difference in per-step cost between two small values of that number.

If the work is a routing decision, a tool call, or a fixed short prompt, this board is the right size. The FunctionGemma build runs a 13-token prompt through cold prefill in 0.48 seconds and completes a turn in 1.3 to 2.0 seconds. That is a real result, and it is achieved by removing the schema from the prompt entirely, not by making the model faster. The prompt is short because the tool names are baked into the weights as single functional tokens.

If the work is a conversation with history, the board is the wrong size. The same project reached for a Mamba hybrid to get around the shape problem, which is an architecture change, not an optimization. For anything that needs a real context window at a usable rate, the entry point is the NVIDIA Jetson Orin Nano Super Developer Kit, listed at $279.99 on Amazon as of October 30, 2026, against a $249 list price. It sat at $399 for most of October, so the current number is the one worth checking on the day you buy. It has 8 GB on a 128-bit LPDDR5 at 102 GB/s, which is roughly sixteen times the bus this board can move. An 8B model at 4-bit weighs about 4.5 GB. ChipRadar’s benchmark write-up puts the practical ceiling for that board in the mid-teens of tokens per second, which is what a 102 GB/s bus actually allows once cache traffic and sampling overhead are counted.

For the sensor side, the Arducam 8MP IMX219 module is the verified-compatible camera for the Jetson carrier board. The linked listing is $23.99 and in stock on Amazon as of October 30, 2026; Arducam’s own store lists the Orin-series IMX219 at $26.99. The Coralboard has a CSI connector, but I did not verify a specific module against that board, so I am not going to recommend one.

The original Coral USB Accelerator is $199.99 on Amazon as of October 30, 2026 — one unit left, from a marketplace seller, not sold by Amazon. Price-history tracking put the third-party price at $135 in late September against a $127.79 list price, so the current number is a single-seller quote rather than a market price. It runs vision models, not language models, and it is not a substitute for this board. It is a collector’s item at this point, and the price reflects that.

For a build that has to serve a context window rather than a single prompt, the tier guidance in the guide to best hardware for running local LLMs still holds, and the reason it holds is the same reason this board cannot hold a 32K conversation: the memory that has to be touched per token is the model itself, and the memory that has to be reserved up front is the shape you promised the compiler.

The $99.79 board is a good instrument for the part of the problem that is a compiler bug. It is not an instrument for the part that is a memory wall, because at this model size there is no memory wall yet. There is a reservation, and the reservation is a policy choice made at compile time. That is the same conclusion as Janas, where the memory wall turns out to be a cache policy problem rather than a capacity problem.

1 Comment

Leave a Reply

Your email address will not be published. Required fields are marked *