Skip to content
anythingai.space
Module 02

How LLMs Work

Tokens, attention, transformers, embeddings and the inference pipeline — from text in to text out.

lessons
6
steps
52
time
74 min
ENCODER · SEES EVERYTHINGThecatsatonmatThecatsatonmatDECODER · SEES ONLY THE PASTThecatsatonmatThecatsatonmatBERT-style → intent, sentiment,embedding modelsGPT-style → text generation, chat,agents
01beginner·12 min·8 steps

LLM Internals

Why transformers replaced RNNs, how Mixture-of-Experts splits memory from compute, and why serving a model is a bandwidth problem.

  • Why self-attention beat recurrence: parallelism and direct long-range links
  • The transformer families and which one powers chat models
  • How MoE separates total parameters (memory) from active parameters (compute)
  • Why labs now over-train small models
Start →
CHARACTER (BYT5) · “RUNNING”runningSUBWORD BPE (GPT) · “RUNNING”runningWORDPIECE (BERT) · “EMBEDDING”em##bed##dingWORD (EARLY NLP)running[UNK]7 tokens — very long sequences2 tokens — the sweet spotunseen words → [UNK]
02beginner·11 min·9 steps

Tokenization Deep Dive

Models read subword tokens, not letters or words — and that one fact drives cost, context limits, multilingual fairness and some famous failures.

  • Why “count the r’s in strawberry” trips models up
  • How BPE learns a vocabulary by merging frequent pairs
  • The vocabulary-size tradeoff and the multilingual tax
  • Budget a context window in tokens with the right tokenizer
Start →
MHA · 8 K/V HEADSK1K2K3K4K5K6K7K8Q1Q2Q3Q4Q5Q6Q7Q8GQA · 2 K/V HEADSK1K2Q1Q2Q3Q4Q5Q6Q7Q8MQA · 1 K/V HEADKQ1Q2Q3Q4Q5Q6Q7Q8KV cache: 100%÷ group size (Llama70B: ÷8)÷ n_heads, some qualityloss
03intermediate·13 min·9 steps

Attention Mechanisms

How tokens look at each other — queries, keys and values, the causal mask, and the head-sharing tricks (GQA, MQA, MLA) that decide your KV-cache bill.

  • Read Q, K and V as search query, index and content
  • Read an attention matrix and apply a causal mask
  • Explain why scores are divided by √dₖ
  • Compare MHA, GQA and MQA by KV-cache size
Start →
TRANSFORMER BLOCK · REPEATED N TIMESresidualresidualInput xRMSNormMasked attentionGQA · RoPE · KV cache+ residualRMSNormFeed-forwardSwiGLU+ residualOutput
04intermediate·12 min·8 steps

Transformer Architecture

One decoder-only transformer, end to end: embeddings, the pre-norm block, RoPE, GQA, SwiGLU and the LM head — and what changed between GPT-2 and 2026.

  • Trace a token from ID to logits
  • Draw a pre-norm transformer block with its two residual paths
  • Explain what changed from GPT-2 to Llama-style models
  • Know when to tie embeddings and why active parameters stopped growing
Start →
BI-ENCODER · FIRST-STAGE RETRIEVALCROSS-ENCODER · RERANKINGDocumentQueryEncoderDoc vectorprecomputedQuery vectorCosinesimilarityQuery + Docone inputCross-encoderRelevanceper pair · O(n)
05beginner·12 min·9 steps

Embeddings & Vector Spaces

Turn text into coordinates where distance means meaning — and learn the dimension, precision and versioning choices that decide what search costs.

  • Picture embeddings as coordinates where nearby means similar
  • Tell bi-encoders (retrieval) from cross-encoders (reranking)
  • Explain contrastive training with positives and hard negatives
  • Shrink vectors with Matryoshka dimensions and quantization
Start →
ONE REQUEST OVER TIMEPrefill4 prompt tokens at onceDecodejumpsoverthelazydogBound bycomputememory bandwidthtime →TTFT ≈ network + queue +prefillTPS = 1 ÷ inter-token latency
06intermediate·14 min·9 steps

The Inference Pipeline

From prompt to streamed answer: why prefill and decode behave so differently, how sampling and speculation work, and which latency and memory numbers to watch.

  • Contrast compute-bound prefill with memory-bound decode
  • Explain how temperature and top-p shape sampling — and why hosted APIs are dropping them
  • Describe speculative decoding and its batch-size catch
  • Compute TTFT, TPS and total latency
Start →
Read deeper

Guide chapters · Foundations

Next module · 03
Choosing Models