Backend for AIintermediate11 min · 10 steps
Async and Await
LLM backends spend most of their time waiting on the network; an event loop lets one thread keep hundreds of those waits in flight.
pre-model latency1,200ms
workers for 40 users40
requestheavy worktokendata
metrics are illustrativeProblemStep 1 of 10
A sync worker spends its life waiting
An LLM backend is I/O-bound. The model takes seconds; retrieval and the ticketing API take hundreds of milliseconds. The CPU does almost nothing in between.
A synchronous worker handles one request at a time. If the average request takes 3.5 s, mostly waiting, that worker serves about 0.3 requests per second. Forty concurrent users need about 40 workers, each one idle most of the time.
Lesson 3 of 20 in Backend for AI