Skip to content
anythingai.space
Module 14

Backend for AI

REST, HTTP, async, streaming, auth, databases, caching, queues, jobs, rate limits, idempotency, retries and observability.

lessons
20
steps
199
time
219 min
POST/v1/conversations201 Created + LocationPOST/v1/conversations/{id}/messages200 text/event-streamGET/v1/conversations/{id}200 JSONGET/v1/kb/articles?q=vpn&cursor=…200 + next_cursorPATCH/v1/tickets/{number}Idempotency-Key200 ticketDELETE/v1/conversations/{id}204 No Content
01beginner·11 min·10 steps

REST APIs

Model your backend as resources at URLs, and let HTTP methods and status codes say what was asked and what happened.

  • Why verb-style endpoints turn into an unmaintainable mess
  • Resources as nouns, methods as intent, status codes as outcome
  • What happens to a request between the client and the database
  • How an AI backend maps chat, streaming and tickets onto REST
Start →
200 {"error":…}→a real 4xx / 5xxretry 400 / 422→fix the requestno timeout→connect + read + deadlineSDK + wrapper + gateway→one retry layer403 → re-login→401 refresh · 403 stopignore Retry-After→max(Retry-After, backoff)
02beginner·10 min·10 steps

HTTP and Status Codes

Status codes are standard outcome signals; a client that reacts to each one correctly survives the 429s, 5xx and timeouts every LLM integration hits.

  • What a request and a response actually carry
  • What each status code family tells the client to do
  • Which failures are worth retrying, and which never are
  • A complete client policy for model calls
Start →
taskstimeout 30 sEvent loop1 threadSemaphore≤ 10 in flightModel APIRetrievalpgvectorServiceNowtickets
03intermediate·11 min·10 steps

Async and Await

LLM backends spend most of their time waiting on the network; an event loop lets one thread keep hundreds of those waits in flight.

  • Why a sync worker wastes most of its time waiting
  • How the event loop switches between tasks at each await
  • Run independent calls concurrently: 1,200 ms → 500 ms
  • The blocking-call mistake that freezes every request
Start →
RESPONSE HEADERS200 OKContent-Type: text/event-streamCache-Control: no-cacheEVENT 1event: statusdata: {"step":"searching articles"}⏎ blank lineevent: tokendata: {"text":"Quit the"}⏎event: citationdata: {"kb":"KB0012345"}⏎: pingcomment = keep-alive⏎event: donedata: {"tokens":284}⏎
04intermediate·11 min·10 steps

Streaming with SSE

Server-sent events stream tokens, agent status and citations over one long-lived HTTP response, which is how model APIs stream and the simplest way to stream to a browser.

  • Why streaming changes how fast an answer feels
  • The SSE wire format: event, data, id, retry and comments
  • Stream status, tokens and citations from one agent turn
  • Cancel upstream work when the user leaves
Start →
auth only at connect→re-auth before expiryno heartbeats→ping / pongLB idle timeout→timeout > ping intervalstate in the pod→Redis + session IDunbounded send queue→bounded queue, drop or closeWebSocket for text chat→SSE
05intermediate·11 min·10 steps

WebSockets

Upgrade an HTTP connection to a persistent full-duplex channel when the client must talk back mid-response: voice, interruptions, live confirmations.

  • Why one-way streaming can't handle an interruption
  • The upgrade handshake and a typed event protocol
  • The interrupt: the reason to choose WebSockets
  • Sizing and scaling 500 concurrent voice sessions
Start →
isshttps://login.example.com/tenant-id/v2.0audapi://it-assistantsubE12345drives every tool callgroupsIT-AllVPN-Usersfilters retrievalscpchat.readchat.writeiat / exp14:00 → 15:001 hour
06intermediate·11 min·10 steps

Authentication with JWT

A JSON Web Token carries signed claims about the caller that every request can verify locally, which is the base for permission-filtered retrieval, user-scoped tools and audit.

  • Why identity must come from a verified token, never the body or the model
  • The three parts of a JWT, and why it is signed but not encrypted
  • The verification checklist every API must run
  • How verified claims drive retrieval filters and tool calls
Start →
scope: *.ReadWrite.All→narrowest scopeapp permission for user data→delegated / OBOno PKCE / state→PKCE + state alwaysrefresh token in localStorage→server-side, rotatedID token as API auth→check access token audredirect URI wildcard→exact registered URIs
07intermediate·12 min·10 steps

OAuth 2.0

OAuth lets an app get limited, expiring tokens to call an API on a user's behalf (or as itself) without ever handling the user's password.

  • The four roles in every OAuth flow
  • Which grant to use for users, services and devices
  • The authorization code + PKCE flow, step by step
  • On-behalf-of: why even a confused agent can't exceed the user
Start →
joins in app code→relational data in Postgressession rows in SQL→Redis with TTLDynamoDB full scans→design keys per queryPDF bytes in a column→object storage + key5 stores on day one→Postgres + Redis first"no schema"→validated models in code
08beginner·10 min·9 steps

SQL vs NoSQL

Relational databases give schemas, joins and transactions; NoSQL families trade some of that for flexible shapes, specific access patterns and scale. Real AI systems use several, chosen by access pattern.

  • The database families and what each is good at
  • Choose by access pattern and consistency, not fashion
  • Map an AI assistant's data onto stores
  • Why NoSQL doesn't mean no schema
Start →
no index on FK→index join + filter cols(created_at, user_id)→(user_id, created_at)12 indexes on one table→drop unusedWHERE lower(email) = …→expression indexLIKE '%vpn%'→full-text / trigramCREATE INDEX (locks)→CREATE INDEX CONCURRENTLY
09intermediate·11 min·10 steps

Database Indexing

An index is a sorted side structure that lets the database jump to the rows a query needs instead of scanning the table: faster reads, paid for with storage and slower writes.

  • Why a query on 2 million rows takes 1.8 seconds without an index
  • How a B-tree finds 20 rows in 3 page reads
  • Composite index column order: equality first, then range or sort
  • Partial, GIN, covering and vector indexes for real queries
Start →
tx open during LLM call→short tx after the callSELECT, then UPDATE→conditional UPDATEsend email inside tx→outbox rowswallow 40001 / deadlock→retry the transactionauto-commit per statement→explicit BEGIN / COMMIT
10intermediate·12 min·10 steps

Transactions and ACID

Group reads and writes into one unit that fully succeeds or fully fails, and use a transactional outbox for the external calls a database transaction can't cover.

  • The double-click race and why check-then-act fails
  • What A, C, I and D each guarantee
  • A transaction with a conditional update that closes the race
  • The transactional outbox for external calls
Start →
session:{conversation}hash30 min idlerl:{user}:{minute}counter60 semb:{sha256(text)}bytes7 daysans:{tenant}:{q-hash}string24 htenant = boundarylock:ticket:{number}NX10 sjobs:ingeststreamuntil consumed
11intermediate·11 min·10 steps

Caching with Redis

Use Redis as a fast cache and coordination layer (session state, cached embeddings and answers, rate-limit counters, locks, queues), always with expiry and never as the only copy of important data.

  • The jobs Redis does in an AI backend
  • The cache-aside pattern, step by step
  • Key naming, TTLs and why scoping is a security boundary
  • Size the savings and the memory: $43/day, ~1.2 GB
Start →
apply twice on redelivery→dedupe by message IDpublish, then commit→outbox, publish afterpoison retried forever→max attempts → DLQ5 MB message→ID + referenceglobal order assumed→key by entityno lag alert→alert on lag + oldest age
12intermediate·12 min·10 steps

Message Queues

Services talk asynchronously through a durable broker, either a work queue where each message goes to one worker, or an event log every consumer group reads at its own pace, to decouple, absorb bursts and retry safely.

  • Why calling every downstream service inline is fragile
  • Work queues: competing consumers, acks, redelivery, dead letters
  • Event logs: partitions, consumer groups, offsets and replay
  • Delivery guarantees and why consumers must be idempotent
Start →
two kb-syncs at once→unique job keyfails silently→alert on fail and no-rundouble-applied on retry→idempotent stepsrestart from zero at 1h55→checkpoint each stepbatch starves users→pools by prioritycron on one VM→managed scheduler + queue
13intermediate·11 min·10 steps

Background Jobs

Run scheduled, triggered and handed-off work outside the request–response cycle, with explicit states, retries, timeouts and enough visibility that failures never go silent.

  • The four kinds of background work
  • The job lifecycle as an explicit state machine
  • Unique keys, heartbeats, checkpoints and priority pools
  • A nightly knowledge sync: one transient failure, one real outage
Start →
counter per replica→central Redis / gatewayrequests only→tokens / costbare 429→Retry-After + headerscapacity 1→bucket sized for burstslimiter down = outage→fail open / closed policyone tier for all→tiers for service accounts
14intermediate·11 min·10 steps

Rate Limiting

Cap requests, tokens or dollars per user, tenant and client so one runaway caller can't take down the service, burn the budget or eat the provider quota, and tell callers when to retry.

  • Why every AI backend needs limits per user, tenant and globally
  • The main algorithms, and why token bucket is the default
  • Simulate a bucket: 16 requests, 11 allowed, 5 rejected
  • Limit LLM endpoints by tokens and concurrent streams, not just requests
Start →
uuid4() per attempt→key per logical opSELECT then INSERT→unique constraint reservekey saved, response not→store status + body5-minute expiry→≥ retry window (24 h)same key, new body: ok→compare request hash"ServiceNow dedupes"→own record + state check
15intermediate·10 min·10 steps

Idempotency Keys

A unique key on each non-idempotent request lets the server recognise a retry and return the original result instead of doing the work twice.

  • Why a lost response after successful work causes duplicates
  • The full key-check sequence, including the replayed 201
  • The server algorithm: reserve, execute, save, replay
  • Deterministic keys for agent tool calls that survive resumes
Start →
retry 422→classify firstsleep(2 ** n)→random(0, 2 ** n)while True: retry→N attempts + deadlineretry POST, no key→Idempotency-Key3 × 3 × 3 = 27→one retry layerretry after user left→propagate the deadline
16intermediate·11 min·10 steps

Retries with Backoff

Re-attempt failed calls after growing, randomised delays so transient failures heal without hammering a struggling dependency or synchronising thousands of clients into a retry storm.

  • Why immediate, synchronised retries make outages worse
  • The rules: which errors, which operations, how many times
  • Exponential backoff with full jitter: 0.5 → 8 s
  • Retrying inside a 6 s time-to-first-token budget
Start →
POST bytes to the API→presigned direct PUTpublic bucket / 7-day URL→private, 5-min URLtrust .pdf→magic bytes2,000-page PDF→25 MB / 300 pagesindex, then scan→scan, then indexkey = ../../name.pdf→tenant/user/uuid
17intermediate·11 min·10 steps

File Uploads to Object Storage

Let clients upload straight to object storage with short-lived presigned URLs your API issues, then validate, scan and process the untrusted file asynchronously.

  • Why proxying uploads through the API is the wrong default
  • The presigned-URL sequence: the API never touches the bytes
  • The async pipeline: scan, verify, parse, chunk, embed, index
  • A ~25 s upload-to-ready timeline and what storage really costs
Start →
?limit=100000→max 100ORDER BY created_at→+ id tiebreakerOFFSET 100000→keyset cursorpage 1 only→follow next linkswhile next: fetch()→safety limitcursor=WHERE id>9→opaque, signedall pages → model→cap + "narrow down"
18beginner·10 min·10 steps

Pagination

Return large result sets in bounded pages, by offset or by cursor, in your own APIs and when your tools consume other people's.

  • Why every list endpoint and every tool needs bounded pages
  • Offset vs cursor, and why deep offsets get slow and unstable
  • Keyset cursors: same speed on every page (240 ms → 3 ms)
  • Paginating external APIs from an agent tool without flooding the context
Start →
payload: dict→typed model, extra=forbid"1" → True→strict fieldstrust model JSON→model_validate_jsonmessage: str→max_length=8000422 + stack trace→field + message onlyhand-written schema→model_json_schema()
19beginner·11 min·10 steps

Input Validation with Pydantic

Declare typed models with constraints and parse every request, model output and tool argument into a validated object, or a precise error, before anything acts on it.

  • Why API inputs and LLM outputs are both untrusted
  • A request model that turns bad input into a precise 422
  • Validate model output, re-ask once, then fall back
  • 24,000 routing decisions a day: 99.7% valid first time
Start →
print("failed")→JSON with fixed schemano request ID→trace_id everywherelog full prompt + email→redact, log referencesdebug on in prod→levels + samplingexcept: pass→log + count the errorCPU-only dashboards→tokens, cost, quality
20intermediate·11 min·10 steps

Logging and Monitoring

Structured logs, metrics and traces, all sharing one trace ID, take you from "something is wrong" to "exactly why" in minutes.

  • The three pillars: logs, metrics and traces
  • Structured JSON log lines and consistent levels
  • Correlation: one trace ID from edge to model call
  • A 429 incident diagnosed in ~10 minutes
Start →
Next module · 15
Deployment & Delivery