Deployment & Deliveryintermediate12 min · 9 steps
GPU Instances
Choose GPU VMs or nodes for self-hosted models by memory, bandwidth, interconnect, availability and price — and size them for weights plus KV cache.
weights140GB
GPUs1
free for KV0GB
requestdata
metrics are illustrativeProblemStep 1 of 9
“The model is 70B. Will it fit?”
The team wants to self-host a 70-billion-parameter model. In BF16 that's 140 GB of weights — already more than a single 80 GB GPU. And weights are only part of it: every concurrent request also needs KV cache memory, which grows with context length.
Size for weights alone and the server either won't load or runs out of memory the moment real traffic arrives.
Watch outOut-of-memory under load is the classic GPU sizing failure: it worked in the demo with one short prompt.
Lesson 10 of 20 in Deployment & Delivery