Skip to content
anythingai.space
How LLMs Workintermediate14 min · 9 steps

The Inference Pipeline

From prompt to streamed answer: why prefill and decode behave so differently, how sampling and speculation work, and which latency and memory numbers to watch.

append & repeatPROMPT, THEN GENERATED TOKENSThequickbrownfoxjumpsoverForward passlogits for next tokenSamplepick one tokenStop?EOS · max · stop seq
tokenheavy workrequest
1 / 9
Key ideaStep 1 of 9

Generation is a loop, one token per lap

LLMs generate autoregressively: run the model over everything so far, get a probability for every vocabulary entry, pick one token, append it, and go again.

The loop stops on an EOS token, a max-tokens limit, or a custom stop sequence. Everything about serving cost and latency comes from how many laps you run and how expensive each lap is.

tokens = tokenize(prompt)
for _ in range(max_tokens):
    logits = model.forward(tokens)
    next_token = sample(logits[-1])
    if next_token == EOS: break
    tokens.append(next_token)

Lesson 6 of 6 in How LLMs Work