Backend for AI
REST, HTTP, async, streaming, auth, databases, caching, queues, jobs, rate limits, idempotency, retries and observability.
- lessons
- 20
- steps
- 199
- time
- 219 min
REST APIs
Model your backend as resources at URLs, and let HTTP methods and status codes say what was asked and what happened.
- Why verb-style endpoints turn into an unmaintainable mess
- Resources as nouns, methods as intent, status codes as outcome
- What happens to a request between the client and the database
- How an AI backend maps chat, streaming and tickets onto REST
HTTP and Status Codes
Status codes are standard outcome signals; a client that reacts to each one correctly survives the 429s, 5xx and timeouts every LLM integration hits.
- What a request and a response actually carry
- What each status code family tells the client to do
- Which failures are worth retrying, and which never are
- A complete client policy for model calls
Async and Await
LLM backends spend most of their time waiting on the network; an event loop lets one thread keep hundreds of those waits in flight.
- Why a sync worker wastes most of its time waiting
- How the event loop switches between tasks at each await
- Run independent calls concurrently: 1,200 ms → 500 ms
- The blocking-call mistake that freezes every request
Streaming with SSE
Server-sent events stream tokens, agent status and citations over one long-lived HTTP response, which is how model APIs stream and the simplest way to stream to a browser.
- Why streaming changes how fast an answer feels
- The SSE wire format: event, data, id, retry and comments
- Stream status, tokens and citations from one agent turn
- Cancel upstream work when the user leaves
WebSockets
Upgrade an HTTP connection to a persistent full-duplex channel when the client must talk back mid-response: voice, interruptions, live confirmations.
- Why one-way streaming can't handle an interruption
- The upgrade handshake and a typed event protocol
- The interrupt: the reason to choose WebSockets
- Sizing and scaling 500 concurrent voice sessions
Authentication with JWT
A JSON Web Token carries signed claims about the caller that every request can verify locally, which is the base for permission-filtered retrieval, user-scoped tools and audit.
- Why identity must come from a verified token, never the body or the model
- The three parts of a JWT, and why it is signed but not encrypted
- The verification checklist every API must run
- How verified claims drive retrieval filters and tool calls
OAuth 2.0
OAuth lets an app get limited, expiring tokens to call an API on a user's behalf (or as itself) without ever handling the user's password.
- The four roles in every OAuth flow
- Which grant to use for users, services and devices
- The authorization code + PKCE flow, step by step
- On-behalf-of: why even a confused agent can't exceed the user
SQL vs NoSQL
Relational databases give schemas, joins and transactions; NoSQL families trade some of that for flexible shapes, specific access patterns and scale. Real AI systems use several, chosen by access pattern.
- The database families and what each is good at
- Choose by access pattern and consistency, not fashion
- Map an AI assistant's data onto stores
- Why NoSQL doesn't mean no schema
Database Indexing
An index is a sorted side structure that lets the database jump to the rows a query needs instead of scanning the table: faster reads, paid for with storage and slower writes.
- Why a query on 2 million rows takes 1.8 seconds without an index
- How a B-tree finds 20 rows in 3 page reads
- Composite index column order: equality first, then range or sort
- Partial, GIN, covering and vector indexes for real queries
Transactions and ACID
Group reads and writes into one unit that fully succeeds or fully fails, and use a transactional outbox for the external calls a database transaction can't cover.
- The double-click race and why check-then-act fails
- What A, C, I and D each guarantee
- A transaction with a conditional update that closes the race
- The transactional outbox for external calls
Caching with Redis
Use Redis as a fast cache and coordination layer (session state, cached embeddings and answers, rate-limit counters, locks, queues), always with expiry and never as the only copy of important data.
- The jobs Redis does in an AI backend
- The cache-aside pattern, step by step
- Key naming, TTLs and why scoping is a security boundary
- Size the savings and the memory: $43/day, ~1.2 GB
Message Queues
Services talk asynchronously through a durable broker, either a work queue where each message goes to one worker, or an event log every consumer group reads at its own pace, to decouple, absorb bursts and retry safely.
- Why calling every downstream service inline is fragile
- Work queues: competing consumers, acks, redelivery, dead letters
- Event logs: partitions, consumer groups, offsets and replay
- Delivery guarantees and why consumers must be idempotent
Background Jobs
Run scheduled, triggered and handed-off work outside the request–response cycle, with explicit states, retries, timeouts and enough visibility that failures never go silent.
- The four kinds of background work
- The job lifecycle as an explicit state machine
- Unique keys, heartbeats, checkpoints and priority pools
- A nightly knowledge sync: one transient failure, one real outage
Rate Limiting
Cap requests, tokens or dollars per user, tenant and client so one runaway caller can't take down the service, burn the budget or eat the provider quota, and tell callers when to retry.
- Why every AI backend needs limits per user, tenant and globally
- The main algorithms, and why token bucket is the default
- Simulate a bucket: 16 requests, 11 allowed, 5 rejected
- Limit LLM endpoints by tokens and concurrent streams, not just requests
Idempotency Keys
A unique key on each non-idempotent request lets the server recognise a retry and return the original result instead of doing the work twice.
- Why a lost response after successful work causes duplicates
- The full key-check sequence, including the replayed 201
- The server algorithm: reserve, execute, save, replay
- Deterministic keys for agent tool calls that survive resumes
Retries with Backoff
Re-attempt failed calls after growing, randomised delays so transient failures heal without hammering a struggling dependency or synchronising thousands of clients into a retry storm.
- Why immediate, synchronised retries make outages worse
- The rules: which errors, which operations, how many times
- Exponential backoff with full jitter: 0.5 → 8 s
- Retrying inside a 6 s time-to-first-token budget
File Uploads to Object Storage
Let clients upload straight to object storage with short-lived presigned URLs your API issues, then validate, scan and process the untrusted file asynchronously.
- Why proxying uploads through the API is the wrong default
- The presigned-URL sequence: the API never touches the bytes
- The async pipeline: scan, verify, parse, chunk, embed, index
- A ~25 s upload-to-ready timeline and what storage really costs
Pagination
Return large result sets in bounded pages, by offset or by cursor, in your own APIs and when your tools consume other people's.
- Why every list endpoint and every tool needs bounded pages
- Offset vs cursor, and why deep offsets get slow and unstable
- Keyset cursors: same speed on every page (240 ms → 3 ms)
- Paginating external APIs from an agent tool without flooding the context
Input Validation with Pydantic
Declare typed models with constraints and parse every request, model output and tool argument into a validated object, or a precise error, before anything acts on it.
- Why API inputs and LLM outputs are both untrusted
- A request model that turns bad input into a precise 422
- Validate model output, re-ask once, then fall back
- 24,000 routing decisions a day: 99.7% valid first time
Logging and Monitoring
Structured logs, metrics and traces, all sharing one trace ID, take you from "something is wrong" to "exactly why" in minutes.
- The three pillars: logs, metrics and traces
- Structured JSON log lines and consistent levels
- Correlation: one trace ID from edge to model call
- A 429 incident diagnosed in ~10 minutes