Skip to content
anythingai.space
Module 05

Inference & Serving

KV cache, batching, speculative decoding, paged attention, serving stacks and the cost playbook.

lessons
9
steps
81
time
112 min
COMPUTE PER BYTE OF MEMORY BANDWIDTHH100 · BF16~300 FLOPs/byteVera Rubin · NVFP4~2,600 FLOPs/byteH100: ~1 PFLOPS dense BF16 ÷ 3.35 TB/sHBM3Rubin: ~50 PFLOPS NVFP4 ÷ 19.2 TB/s HBM4Decode reads the weights once per step — so batch many sequences(or verify several speculative tokens) to use the compute you paidfor
01beginner·11 min·9 steps

Inference Fundamentals

LLM inference is two jobs with opposite bottlenecks — compute-bound prefill and memory-bound decode — and every optimization targets one of them.

  • Why generation is slower than classification
  • Prefill is compute-bound, decode is memory-bound
  • The memory wall: why decode needs big batches
  • TTFT, TPOT and goodput — and why batch-1 numbers mislead
Start →
KV CACHE: LAYERS × TOKENS SO FARt1t2t3t4t5t6t7t8t9layer 1layer 2layer 3…layer 80Each new token adds a column ofK and V to every layer…and the whole grid is repeatedfor every concurrent user
02intermediate·13 min·9 steps

KV Cache and Context Caching

The KV cache is the biggest memory consumer in long-context serving — shrink it with architecture, reuse it across requests, spill it to cheaper tiers, and pay 0.1× for it through API caching.

  • Size the KV cache: ~42 GB per user at 128K for a 70B GQA model
  • How GQA, MQA and MLA shrink KV
  • Reuse shared prefixes and spill KV to cheaper tiers
  • API prompt caching break-even and the hit-rate trap
Start →
1 · DRAFT (5 × 5 MS)The·refund·window·is·302 · TARGET VERIFIES ALL 5 IN ONE 50 MS PASSTheaccept·refundaccept·windowaccept·isaccept·30reject3 · EMITTED THIS ROUNDThe·refund·window·is·14target's token
03intermediate·12 min·9 steps

Speculative Decoding

Guess several tokens cheaply, verify them all in one pass of the big model, and get 2–3× faster decode with exactly the same output.

  • Why one token per weight read wastes the GPU
  • The draft → verify → accept loop
  • Why the output is provably unchanged
  • Which drafters to use in 2026
Start →
long promptshort → bypassKV · RDMA / NVLink10K-token prompt ≈ 3 GBof KV ≈ 65 ms at 400Gb/sRequestsRouterPrefill poolcompute-boundDecode poolmemory-boundStreamtokens out
04intermediate·13 min·9 steps

Batching Strategies

From static batches that wait for the slowest request to continuous batching, chunked prefill and fully disaggregated prefill/decode pools.

  • Why static batching wastes the GPU on LLMs
  • Continuous batching: join and leave every iteration
  • Admission control so queues don't destroy TTFT
  • Chunked prefill to stop long prompts stalling decode
Start →
LOGICAL — REQUEST A SEES ONE CONTIGUOUS SEQUENCEA0tok 0–15A1tok 16–31A2tok 32–47PHYSICAL — BLOCKS WHEREVER VRAM IS FREEB0P0A0P1freeP2A2P3B1P4A1P5freeP6B2P7freeP8freeP9Block tablelogical → physical
05intermediate·11 min·9 steps

PagedAttention

Borrow virtual memory from operating systems: store the KV cache in small fixed-size blocks, cut memory waste from 60–80% to under 4%, and share prefixes for free.

  • Internal vs external fragmentation in contiguous KV allocation
  • Logical vs physical blocks and the block table
  • Why less waste means bigger batches and more throughput
  • Prefix sharing with copy-on-write and block hashing
Start →
ONE REQUEST THROUGH AN 80-LAYER MODEL ON 2 GPUSTP · GPU 1layers 1–80 · half of eachTP · GPU 2layers 1–80 · other halfPP · GPU 1layers 1–40idle (bubble)PP · GPU 2idle (bubble)layers 41–80time →TP: per-layer all-reduce over NVLinkEP + data-parallel attention:standard for big MoE
06advanced·14 min·9 steps

Serving Infrastructure

From one GPU box to an inference fleet: gateways, KV-aware routing, model parallelism, autoscaling, streaming failover and an engine chosen per workload.

  • What an inference gateway does, and why it's a prime target
  • Route by KV-cache overlap, not sticky sessions
  • Tensor vs pipeline vs expert vs context parallelism
  • Fail before the first token, not after
Start →
PRICE MULTIPLIER VS STANDARDCache read0.1× (0.05×, 0.025×)Batch / Flex0.5×Off-peak0.5×Standard1×Priority / Fast~1.75–2.5×Ultrafast6×Offline jobs → Batch 0.5×Interactive → StandardPremium latency → Fast /Ultrafast
07intermediate·13 min·9 steps

Cost Optimization Playbook

Cut inference cost by routing to the cheapest capable model, keeping prefixes cached, picking a service tier per workload and knowing when self-hosting pays.

  • Model cascading: send 90% of traffic to the cheapest capable tier
  • Why caching and prompt length break the tier price order
  • Choose a service tier per workload class
  • Input tokens and reasoning effort drive agent cost
Start →
BLOCKS OF 4: LEFT TO RIGHT, PARALLEL INSIDEdefblock 1add(a,block 1b):block 1returnblock 1ablock 2[MASK]block 2bblock 2[MASK]block 2[MASK]block 3[MASK]block 3block size 1 → pure ARin between → KV cache +variable lengthwhole sequence → purediffusion
08advanced·12 min·9 steps

Diffusion Language Models

Instead of writing left to right, start from a fully masked sequence and denoise every position in parallel — fast, good at code and infill, still behind on hard reasoning.

  • How masked diffusion differs from autoregressive decoding
  • Training by masking and sampling by confidence-based unmasking
  • The speed-vs-quality knob in the unmasking schedule
  • Block diffusion and diffusion drafters for AR models
Start →
LOCAL RUNTIME LAYERSExperience · Ollama · LM StudioEngine · llama.cpp · MLXServing · vLLM · SGLang · vllm-metalEmbedded · ExecuTorch · Core ML · ONNXprototype: one-command pull & runportable inference, GGUF / Appleproduction: concurrent,batchedphones, wearables, microcontrollers
09intermediate·13 min·9 steps

On-Device and Edge Deployment

Run models on laptops, phones and edge boxes: know the runtime layers, why Ollama isn't a server, how quantization makes models fit, and when local beats an API.

  • When local or edge inference beats a cloud API
  • The runtime stack: experience layer, engine, server, embedded
  • Why Ollama serializes and vLLM batches
  • Size memory: weights by quant, plus KV and headroom
Start →
Read deeper

Guide chapters · Inference Optimization

Next module · 06
Prompting & Context