Inference & Servingbeginner11 min · 9 steps
Inference Fundamentals
LLM inference is two jobs with opposite bottlenecks — compute-bound prefill and memory-bound decode — and every optimization targets one of them.
TTFT400ms
TPOT50ms
weights140GB
requesttokendataheavy work
metrics are illustrativeProblemStep 1 of 9
Why does generation feel slow?
A classifier reads the whole input and answers in one parallel pass. An LLM can read your prompt the same way — but it has to write its answer one token at a time.
Generation is autoregressive: each new token depends on the ones before it, so the model runs a sequential loop. One forward pass per output token.
InsightReading the prompt and writing the answer are two different workloads. Treating them as one is the root of most serving confusion.
Lesson 1 of 9 in Inference & Serving