Skip to content
anythingai.space
Inference & Servingintermediate13 min · 9 steps

KV Cache and Context Caching

The KV cache is the biggest memory consumer in long-context serving — shrink it with architecture, reuse it across requests, spill it to cheaper tiers, and pay 0.1× for it through API caching.

KV @128K42GB
TTFT1,500ms
prefix hit0%
KV CACHE: LAYERS × TOKENS SO FARt1t2t3t4t5t6t7t8t9layer 1layer 2layer 3…layer 80Each new token adds a column ofK and V to every layer…and the whole grid is repeatedfor every concurrent user
requestdata
metrics are illustrative
1 / 9
ProblemStep 1 of 9

Every token leaves K and V behind

To generate the next token, attention needs the Key and Value vectors of every previous token, in every layer. Recomputing them each step would be absurd, so the engine keeps them: the KV cache.

It grows by one column per token, in every layer, for every concurrent user. In long-context systems it becomes the single largest consumer of GPU memory.

InsightManaging this cache is the difference between a system that scales to 1M+ tokens and one that falls over at 10K.

Lesson 2 of 9 in Inference & Serving