Tokens & embeddings: cost, limits, and meaning-as-math

Tokens are the atoms of everything you'll build: they set your costs, your limits, and your latency. Embeddings are how meaning becomes math. Master both vocabularies now — every later lesson uses them.

Tokens: the billing and context unit

Models don't see words — they see tokens, subword chunks from a fixed vocabulary (~50–200k entries). Rules of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word; code and non-English text tokenize less efficiently.

"CrackKit sells interview prep bundles."
→ ["Crack", "Kit", " sells", " interview", " prep", " bundles", "."]  (7 tokens)

Three numbers tokens control:

1. 1Cost — APIs bill per input token + per output token (output usually 3–5× pricier).
2. 2Context window — the hard cap on input + output tokens per request (128k–1M+ depending on model). Long chats, big documents, and RAG results all compete for this budget.
3. 3Latency — output tokens are generated one at a time; long answers are slow answers. Input is processed in parallel and is much faster.

Embeddings: meaning as coordinates

An embedding model (separate from the chat model) maps text → a vector of floats (~256–3072 dims) where semantic similarity = geometric closeness:

Similarity is measured with cosine similarity (angle between vectors, 1 = identical direction). This one primitive powers:

• Semantic search — find documents by meaning, not keywords (the heart of RAG)
• Clustering / dedup — group similar tickets, reviews, questions
• Classification — nearest-labeled-neighbor, no training needed
• Recommendations — "users who liked things near this vector..."

similarity = dot(a, b) / (norm(a) * norm(b))   # cosine similarity

The two-model architecture to internalize

Almost every AI product uses two different models: an embedding model to find relevant text, and a chat model to reason over it. Cheap-and-fast finds; smart-and-expensive answers. RAG (section 4) is exactly this pipeline.