How LLMs Workintermediate12 min · 8 steps
Transformer Architecture
One decoder-only transformer, end to end: embeddings, the pre-norm block, RoPE, GQA, SwiGLU and the LM head — and what changed between GPT-2 and 2026.
datatoken
Key ideaStep 1 of 8
Text in, next-token scores out
A decoder-only transformer — the shape behind GPT-3, Llama and most open models — is a short pipeline. Token IDs look up vectors in an embedding table, flow through N identical blocks, get a final RMSNorm, and the LM head projects each vector to one score per vocabulary entry: the logits.
In training, logits at every position feed a cross-entropy loss. In generation only the last position matters: sample one token, append it, repeat.
InsightGPT, Claude and Gemini configurations aren't public; the decoder-only shape is widely assumed for closed frontier models.
Lesson 4 of 6 in How LLMs Work