Inference & Servingintermediate13 min · 9 steps
KV Cache and Context Caching
The KV cache is the biggest memory consumer in long-context serving — shrink it with architecture, reuse it across requests, spill it to cheaper tiers, and pay 0.1× for it through API caching.
KV @128K42GB
TTFT1,500ms
prefix hit0%
requestdata
metrics are illustrativeProblemStep 1 of 9
Every token leaves K and V behind
To generate the next token, attention needs the Key and Value vectors of every previous token, in every layer. Recomputing them each step would be absurd, so the engine keeps them: the KV cache.
It grows by one column per token, in every layer, for every concurrent user. In long-context systems it becomes the single largest consumer of GPU memory.
InsightManaging this cache is the difference between a system that scales to 1M+ tokens and one that falls over at 10K.
Lesson 2 of 9 in Inference & Serving