Skip to content
anythingai.space
Inference & Servingintermediate12 min · 9 steps

Speculative Decoding

Guess several tokens cheaply, verify them all in one pass of the big model, and get 2–3× faster decode with exactly the same output.

per token50ms
tokens / pass1
all weightsPLAIN DECODETargett1t2t3t4t550 ms per pass → 5 tokens in 250 msHBM140 GB weightsTarget · 70B50 ms per passOutput1 token per pass
tokenheavy workerror
metrics are illustrative
1 / 9
ProblemStep 1 of 9

140 GB of reads for one 2-byte token

Decode is memory-bound. A 70B model streams ~140 GB of weights out of HBM to produce one token per sequence — and then does it all again for the next one.

The arithmetic units sit mostly idle. A pass that checked five tokens would cost about the same as a pass that produced one.

Lesson 3 of 9 in Inference & Serving