How LLMs Work
Tokens, attention, transformers, embeddings and the inference pipeline — from text in to text out.
- lessons
- 6
- steps
- 52
- time
- 74 min
LLM Internals
Why transformers replaced RNNs, how Mixture-of-Experts splits memory from compute, and why serving a model is a bandwidth problem.
- Why self-attention beat recurrence: parallelism and direct long-range links
- The transformer families and which one powers chat models
- How MoE separates total parameters (memory) from active parameters (compute)
- Why labs now over-train small models
Tokenization Deep Dive
Models read subword tokens, not letters or words — and that one fact drives cost, context limits, multilingual fairness and some famous failures.
- Why “count the r’s in strawberry” trips models up
- How BPE learns a vocabulary by merging frequent pairs
- The vocabulary-size tradeoff and the multilingual tax
- Budget a context window in tokens with the right tokenizer
Attention Mechanisms
How tokens look at each other — queries, keys and values, the causal mask, and the head-sharing tricks (GQA, MQA, MLA) that decide your KV-cache bill.
- Read Q, K and V as search query, index and content
- Read an attention matrix and apply a causal mask
- Explain why scores are divided by √dₖ
- Compare MHA, GQA and MQA by KV-cache size
Transformer Architecture
One decoder-only transformer, end to end: embeddings, the pre-norm block, RoPE, GQA, SwiGLU and the LM head — and what changed between GPT-2 and 2026.
- Trace a token from ID to logits
- Draw a pre-norm transformer block with its two residual paths
- Explain what changed from GPT-2 to Llama-style models
- Know when to tie embeddings and why active parameters stopped growing
Embeddings & Vector Spaces
Turn text into coordinates where distance means meaning — and learn the dimension, precision and versioning choices that decide what search costs.
- Picture embeddings as coordinates where nearby means similar
- Tell bi-encoders (retrieval) from cross-encoders (reranking)
- Explain contrastive training with positives and hard negatives
- Shrink vectors with Matryoshka dimensions and quantization
The Inference Pipeline
From prompt to streamed answer: why prefill and decode behave so differently, how sampling and speculation work, and which latency and memory numbers to watch.
- Contrast compute-bound prefill with memory-bound decode
- Explain how temperature and top-p shape sampling — and why hosted APIs are dropping them
- Describe speculative decoding and its batch-size catch
- Compute TTFT, TPS and total latency