Skip to content
anythingai.space
Inference & Servingbeginner11 min · 9 steps

Inference Fundamentals

LLM inference is two jobs with opposite bottlenecks — compute-bound prefill and memory-bound decode — and every optimization targets one of them.

TTFT400ms
TPOT50ms
weights140GB
prompttokens, one by onePROMPT — READ IN ONE PARALLEL PASSSummarise·this·contract·for·meANSWER — ONE FORWARD PASS PER TOKENThepass 1·contractpass 2·setspass 3·…pass 4UserLLM · 70Bautoregressive
requesttokendataheavy work
metrics are illustrative
1 / 9
ProblemStep 1 of 9

Why does generation feel slow?

A classifier reads the whole input and answers in one parallel pass. An LLM can read your prompt the same way — but it has to write its answer one token at a time.

Generation is autoregressive: each new token depends on the ones before it, so the model runs a sequential loop. One forward pass per output token.

InsightReading the prompt and writing the answer are two different workloads. Treating them as one is the root of most serving confusion.

Lesson 1 of 9 in Inference & Serving