Inference & Servingintermediate11 min · 9 steps
PagedAttention
Borrow virtual memory from operating systems: store the KV cache in small fixed-size blocks, cut memory waste from 60–80% to under 4%, and share prefixes for free.
KV wasted70%
requests / GPU4
requestdata
metrics are illustrativeProblemStep 1 of 9
Reserve for the longest possible answer
Standard frameworks allocate one big contiguous buffer per request. Since you don't know how long the answer will be, you reserve for max_sequence_length — say 8,192 tokens.
If the user's answer is 10 tokens, 99.9% of that reservation is dead weight. That's internal fragmentation.
Lesson 5 of 9 in Inference & Serving