Skip to content
anythingai.space
Backend for AIintermediate11 min · 10 steps

Async and Await

LLM backends spend most of their time waiting on the network; an event loop lets one thread keep hundreds of those waits in flight.

pre-model latency1,200ms
workers for 40 users40
ONE SYNC WORKER, THREE REQUESTS (SECONDS)request Awaiting on modelrequest Bqueuedwaitingrequest Cqueuedwaitingtime →3.5 s per request → ~0.3 req/s per worker → ~40 workers for 40users
requestheavy worktokendata
metrics are illustrative
1 / 10
ProblemStep 1 of 10

A sync worker spends its life waiting

An LLM backend is I/O-bound. The model takes seconds; retrieval and the ticketing API take hundreds of milliseconds. The CPU does almost nothing in between.

A synchronous worker handles one request at a time. If the average request takes 3.5 s, mostly waiting, that worker serves about 0.3 requests per second. Forty concurrent users need about 40 workers, each one idle most of the time.

Lesson 3 of 20 in Backend for AI