An LLM is a program that predicts the next piece of text. That single function, scaled up, produces summarisation, code generation, translation, and something that behaves like reasoning. This covers how it works, and which parts of it decide what you can actually run on your own hardware.
How an LLM works
An LLM is trained on a large text corpus — books, web pages, code — and learns statistical relationships between tokens. At inference it does one thing repeatedly: given the tokens so far, it produces a probability distribution over the next token. Sampling picks one, appends it, and the process repeats.
That is not understanding in the human sense. What looks like reasoning is a consequence of scale. A model with enough parameters and enough data develops internal representations that generalise to tasks it was never explicitly taught. Summarisation is not programmed into the model; it emerges from training.
The four numbers that matter
- Parameters: the internal variables learned during training. More capacity, and more memory required to hold them.
- Context window: how many tokens the model can attend to at once. This is a memory cost, not a quality cost. A bigger window means a bigger KV cache.
- Quantisation: storing weights at fewer bits per parameter. This is what makes a 27B model fit on a consumer card.
- Temperature: how peaked the sampling distribution is. It controls variability, not intelligence.
Quantisation is why local models exist at all
At full precision a model needs two bytes per parameter. A 27B model is 54 GB of weights before anything else loads. At 4-bit quantisation the same model is about 14 GB, which is why a 24 GB card can run it. The trade is not free: aggressive quantisation degrades output, and the degradation is uneven. Coding and arithmetic degrade faster than conversational fluency.
Approximate weight memory for the sizes people actually run:
| Model size | FP16 | Q4 | Practical tier |
|---|---|---|---|
| 7B | 14 GB | 3.5 GB | Any 8 GB card, most laptops |
| 14B | 28 GB | 7 GB | 12 GB card |
| 27B | 54 GB | 14 GB | 24 GB card, or 32 GB unified memory |
| 35B MoE, 3B active | 70 GB | 18 GB | 24 GB card with offload, or 32 GB unified memory |
Weights are only part of the requirement. The KV cache grows with context length, and a long context on a 27B model adds several gigabytes on top of the quantised weights. This is why a model that fits on paper runs out of memory in practice. The memory floor for each model size, and which card or unified-memory tier covers it, is worked out in the hardware guide. A 27B model at 4-bit sits just inside a 24 GB card; a 35B MoE does not, and that is the point where the build stops being a hobby build.
Dense and MoE models need different hardware
A dense model uses every parameter for every token. A mixture-of-experts model activates a small subset. A 125B MoE that activates 6B per token needs memory for all 125B but compute for only 6B, which is why it runs on a card that could never hold a dense 125B model. The split is what makes engines like Strata work: weights spread across GPU, system RAM, and SSD, with only the active experts kept hot.
What LLMs are actually good at
Tasks with verifiable structure: code, extraction, translation, summarising long documents, classification. Tasks where they are weaker: arithmetic, counting, and exact recall of a specific fact from a long context. The failure mode is not ignorance, it is confidence. A model states a wrong answer with the same certainty as a right one, so anything where correctness matters needs a check you can perform independently.
Running one locally
The fastest way to understand the limits is to run a model. A Mac mini M4 runs quantised 7-14B models through MLX with no setup. On a PC, any spare GPU handles beginner-sized models, and the runtime choice — llama.cpp, Ollama, vLLM — changes throughput more than the model choice does. That comparison is in the backend battle.
If you are building, a 64GB DDR5 kit gives more headroom than a beginner uses, because at 7B the binding constraint is the card, not the memory. You hit the memory wall when you move up model sizes, not before.
For what to run at each size, see the open-source guide.


Pingback: The 2026 LLM Backend Battle: vLLM vs. Ollama vs. llama.cpp - GenX AI Tools
Pingback: Best Open-Source LLMs for Local Deployment (September 2026) - GenX AI Tools
Pingback: AI in the Backyard: Is Lidar-Driven Automation (RockMow X120H) the Future? - GenX AI Tools