Inference & Serving
KV cache, batching, speculative decoding, paged attention, serving stacks and the cost playbook.
- lessons
- 9
- steps
- 81
- time
- 112 min
Inference Fundamentals
LLM inference is two jobs with opposite bottlenecks — compute-bound prefill and memory-bound decode — and every optimization targets one of them.
- Why generation is slower than classification
- Prefill is compute-bound, decode is memory-bound
- The memory wall: why decode needs big batches
- TTFT, TPOT and goodput — and why batch-1 numbers mislead
KV Cache and Context Caching
The KV cache is the biggest memory consumer in long-context serving — shrink it with architecture, reuse it across requests, spill it to cheaper tiers, and pay 0.1× for it through API caching.
- Size the KV cache: ~42 GB per user at 128K for a 70B GQA model
- How GQA, MQA and MLA shrink KV
- Reuse shared prefixes and spill KV to cheaper tiers
- API prompt caching break-even and the hit-rate trap
Speculative Decoding
Guess several tokens cheaply, verify them all in one pass of the big model, and get 2–3× faster decode with exactly the same output.
- Why one token per weight read wastes the GPU
- The draft → verify → accept loop
- Why the output is provably unchanged
- Which drafters to use in 2026
Batching Strategies
From static batches that wait for the slowest request to continuous batching, chunked prefill and fully disaggregated prefill/decode pools.
- Why static batching wastes the GPU on LLMs
- Continuous batching: join and leave every iteration
- Admission control so queues don't destroy TTFT
- Chunked prefill to stop long prompts stalling decode
PagedAttention
Borrow virtual memory from operating systems: store the KV cache in small fixed-size blocks, cut memory waste from 60–80% to under 4%, and share prefixes for free.
- Internal vs external fragmentation in contiguous KV allocation
- Logical vs physical blocks and the block table
- Why less waste means bigger batches and more throughput
- Prefix sharing with copy-on-write and block hashing
Serving Infrastructure
From one GPU box to an inference fleet: gateways, KV-aware routing, model parallelism, autoscaling, streaming failover and an engine chosen per workload.
- What an inference gateway does, and why it's a prime target
- Route by KV-cache overlap, not sticky sessions
- Tensor vs pipeline vs expert vs context parallelism
- Fail before the first token, not after
Cost Optimization Playbook
Cut inference cost by routing to the cheapest capable model, keeping prefixes cached, picking a service tier per workload and knowing when self-hosting pays.
- Model cascading: send 90% of traffic to the cheapest capable tier
- Why caching and prompt length break the tier price order
- Choose a service tier per workload class
- Input tokens and reasoning effort drive agent cost
Diffusion Language Models
Instead of writing left to right, start from a fully masked sequence and denoise every position in parallel — fast, good at code and infill, still behind on hard reasoning.
- How masked diffusion differs from autoregressive decoding
- Training by masking and sampling by confidence-based unmasking
- The speed-vs-quality knob in the unmasking schedule
- Block diffusion and diffusion drafters for AR models
On-Device and Edge Deployment
Run models on laptops, phones and edge boxes: know the runtime layers, why Ollama isn't a server, how quantization makes models fit, and when local beats an API.
- When local or edge inference beats a cloud API
- The runtime stack: experience layer, engine, server, embedded
- Why Ollama serializes and vLLM batches
- Size memory: weights by quant, plus KV and headroom