Transformers & attention, without the calculus
You'll never implement a transformer at work — but interviewers ask about it, model behavior makes more sense when you get it, and "attention" vocabulary is everywhere. Here's the working intuition, no matrix calculus.
The problem attention solves
Older models (RNNs) read text like a human with amnesia — word by word, squeezing everything into one running memory that faded over distance. Transformers instead let every token look directly at every other token, no matter how far apart.
Attention in one sentence
For each token, attention asks: "which other tokens in this text matter for understanding me, and by how much?" — then blends their information in, weighted by relevance.
Mechanically: each token emits a query ("what am I looking for?"), a key ("what am I about?"), and a value ("what I contribute"). Attention scores = how well queries match keys; output = values blended by score. Multi-head = many of these lookups in parallel, each head learning a different relationship (syntax, references, position...).
The full stack, top to bottom
- 1Tokens → embedding vectors (+ positional info so order matters)
- 2Dozens of identical layers: attention (gather context) → feed-forward network (process it)
- 3Final vector → scores over the vocabulary → next-token probabilities
That's the whole architecture. "175B parameters" = the learned weights inside those attention and feed-forward blocks.
Why this explains behaviors you'll see in production
| Behavior | Transformer reason |
|---|---|
| Quadratic cost of long contexts | attention compares every token pair: n² |
| "Lost in the middle" — mid-context info gets ignored | attention concentrates on start + end of long inputs |
| Prompt order matters | attention is position-aware; instructions at the end often win |
| KV-cache makes long chats affordable | keys/values of old tokens are cached, not recomputed |
| Output degrades near context limit | less room for attention to organize generation |
Practical takeaway: put critical instructions at the start or end of long prompts, never buried in the middle — this isn't folklore, it's the attention pattern.