How LLMs actually work (the honest mental model)
You don't need a PhD to build with LLMs — but you do need an accurate mental model. Most production bugs in AI apps come from developers holding the wrong one.
The honest one-liner
An LLM is a function that takes a sequence of tokens and outputs a probability distribution over the next token. That's it. Everything else — chat, reasoning, agents — is scaffolding built on repeatedly sampling "what token comes next?"
What "training" gave the model
- Pretraining — predict the next token across trillions of tokens of text. Result: a compressed statistical model of language, facts, code, and reasoning patterns.
- Post-training (instruction tuning + RLHF/RLAIF) — teach it to behave like an assistant: follow instructions, refuse harm, format answers. This is why "chat" models answer questions instead of continuing your sentence.
Consequences you must design around
| Model property | Engineering consequence |
|---|---|
| Predicts plausible text, not truth | Hallucinations — plausible ≠ correct. Ground with RAG (section 4) or tools |
| No memory between calls | You manage state: send conversation history every request |
| Fixed training cutoff | Fresh data must arrive via the prompt or tools |
| Sees tokens, not letters | Struggles with character-level tasks ("count the r's in strawberry") |
| Sampling is stochastic | Same input can give different outputs — design for it, or pin temperature 0 |
The mental-model test: if a feature of your app assumes the model "knows" or "remembers" something you didn't put in the current request, the design is broken. The prompt is the entire universe the model sees.
Why they still feel smart
Scale. With enough parameters and data, next-token prediction forces the model to internalize grammar, world knowledge, code semantics, and reasoning-shaped patterns — because predicting text written by reasoning humans requires approximating reasoning. "It's just autocomplete" undersells it; "it thinks like a person" oversells it. Hold both.