Skip to content
anythingai.space
Module 15

Deployment & Delivery

Docker, Kubernetes, CI/CD, eval gates, IaC, GPUs, autoscaling, canaries, flags, versioning, health checks and rollbacks.

lessons
20
steps
186
time
221 min
NAIVE: CODE COPIED BEFORE DEPSpip install (all deps) — rebuiltCOPY . . — changedpython:3.12 full base + dev toolsOPTIMIZED: DEPS FIRST, CODE LASTCOPY app/ — rebuilt (small)pip install — cachedpython:3.12-slim — cached1-line change → reinstall everything1-line change → one smalllayer
01beginner·11 min·9 steps

Docker

Package the app and its exact runtime into one immutable image that runs the same on a laptop, in CI and in production.

  • Why fragile LLM stacks need an immutable image
  • Dockerfile, image, container and registry — and how layers are cached
  • Cut a RAG service image from ~1.6 GB to ~350 MB and rebuilds from 4 min to 15 s
  • The mistakes that leak secrets or cut off streams
Start →
COMPOSE NETWORK (PRIVATE)redis:6379db:5432jobswritesollama:11434tracesEngineerlocalhost:8000apiFastAPI · :8000→hostredissessions · queuedbPostgres + pgvectorworkerqueue consumerollamaoptional profilelangfusetracing · optional
02beginner·10 min·9 steps

Docker Compose

Describe the API, worker, database, cache, local model and tracing in one YAML file and start the whole stack with one command.

  • Why hand-installing a RAG stack costs a day and drifts
  • Services, networks, volumes and profiles on one private network
  • Health checks and depends_on so services start in the right order
  • Where Compose stops and an orchestrator takes over
Start →
CPU NODE POOL · API, WORKERS, GATEWAYSGPU POOL (TAINTED)scalesIngressTLS · /apiServicestable name + VIPHPA3–12 podsConfigMap + Secretvault-backedPod 1cpu 1 · mem 2GiPod 2cpu 1 · mem 2GiPod 3cpu 1 · mem 2GiWorker podsingestion jobsvLLM podtoleration: gpu
03intermediate·13 min·9 steps

Kubernetes Basics

Declare what should run — image, replicas, resources, how to reach it — and let the orchestrator schedule, restart, scale and roll it out.

  • Desired state and the reconciliation loop
  • The core objects: Pod, Deployment, Service, Ingress, ConfigMap/Secret, HPA, node pools
  • How a rolling update and probes keep traffic on healthy pods
  • Size the rag-api: 200 concurrent ÷ 40 per pod 5 pods + headroom
Start →
CI · EVERY PULL REQUESTCD · AFTER MERGEsame imagePull requestcode · prompt · configCItests · build · evalsImage @ SHAbuilt onceStagingE2E testsCanary 5%errors · p95 · qualityProduction 100%Release recordwhat shipped
04intermediate·12 min·9 steps

CI/CD Pipelines

Verify every change — including prompts, models and retrieval settings — and release it in safe, gradual, reversible steps.

  • Why LLM systems change through prompts, models and data — not just code
  • The CI stages for a pull request, including fast retrieval checks and an eval gate
  • CD: same image → staging → canary → 100%, with automatic rollback and a release record
  • Read the DORA metrics before and after
Start →
matrixmain onlytestlint · unit · cachebuildimage @ SHAeval-knowledgegolden sliceeval-actiontrajectoriesreportPR commentdeploy-stagingmain only · protected
05intermediate·11 min·9 steps

GitHub Actions

Run tests, image builds and parallel eval suites as YAML workflows in the repo — fast, cheap per run and locked down.

  • Workflows, events, jobs, steps and runners
  • An LLM pipeline: test → build → eval matrix → report → deploy-staging
  • Short-lived cloud credentials with OIDC instead of stored keys
  • Cut a 30-minute serial eval to ~5 minutes with a matrix
Start →
passpasspromotedev golden setGate 1held-out + red-teamGate 2live metrics vs controlGate 3Changecode · prompt · modelGate 1pre-mergeGate 2pre-productionGate 3canary 5%EveryoneBlockedfix and re-run
06intermediate·12 min·9 steps

Eval Gates in CI

Turn eval suites into required checkpoints that block a change — code, prompt, model or index — from advancing when quality, safety, cost or latency slip.

  • Why an eval nobody must pass is advice, not a control
  • The three gates: pre-merge, pre-production and canary
  • Write gate policies as code with baselines, thresholds and slices
  • Follow a cheaper model through all three gates — and see where it gets blocked
Start →
DEVSTAGINGPRODUCTIONDev modelshared quotakb-devseed dataMock ServiceNowProd modelstaging quotakb-stagingredacted copySub-prod SNProd model2 regionskb-prodProd ServiceNow
07intermediate·11 min·10 steps

Environment Configs

Keep what differs between dev, staging and production out of the code and the image, so one tested image behaves correctly everywhere.

  • How a silent default sent staging tests to production
  • Classify every setting: secret, environment config, flag or constant
  • What differs between dev, staging and production for an LLM assistant
  • Typed, validated config with clear precedence
Start →
REGION AREGION B (NEW)primaryfailoverTIME TO ADD A REGION (ILLUSTRATIVE)Portal~1 dayIaC< 1 hModel deploymentprimaryModel deployment+ quota · private endpoint+4rag-api~ 2nd endpoint
08intermediate·11 min·10 steps

Infrastructure as Code

Define model deployments, search indexes, clusters, vaults and permissions in versioned files that are planned, reviewed and applied by a pipeline.

  • Why portal clicking doesn't scale: drift, no review, no rebuild
  • The IaC loop: declare → PR → plan + policy → merge → pipeline apply → drift check
  • Modules per environment and remote, locked state
  • Add a second-region model deployment in under an hour
Start →
GOOD FITSPOOR FITSParse + embed uploadWebhook → summariseNightly re-indexSaaS event glueLong agent runsStreaming chatGPU model servingSteady high traffic
09beginner·10 min·9 steps

Serverless Functions

Run small pieces of code on events — uploads, queue messages, schedules, webhooks — that scale from zero and bill per execution, and know where they don't fit.

  • Triggers, instances that scale from zero, and per-execution billing
  • Cold starts, time limits, retries and why handlers must be idempotent
  • Good and poor fits for LLM systems
  • Cost out ingestion vs a chat API on functions
Start →
BF16 ON 2× 80 GB (MINIMUM)Weights 140 GBKV 20FP8 ON 1× 80 GBWeights 70 GBKV4-BIT ON 1× 80 GBWeights ~38KV + headroom ~42 GBRealistically 4 GPUs for KV headroom + throughput~10 GB KV → limited concurrencyPlenty of KV · check quality ongolden set
10intermediate·12 min·9 steps

GPU Instances

Choose GPU VMs or nodes for self-hosted models by memory, bandwidth, interconnect, availability and price — and size them for weights plus KV cache.

  • Why sizing by weights alone ends in out-of-memory errors
  • What matters in a GPU: memory, bandwidth, precision, interconnect, price
  • Weights parameters × bytes per parameter, plus 20–40% for KV cache
  • Fit a 70B model: BF16 vs FP8 vs 4-bit
Start →
TERMINALollama pull llama3.1:8bollama pull nomic-embed-textollama run llama3.1:8bollama psMODEL MEMORY OVER TIMEchat 8Bloadin memory · servingidleembedderindexingminutes →chat model unloads after idle
11beginner·9 min·9 steps

Ollama for Local Models

Run open-weight chat and embedding models on your own machine behind an OpenAI-compatible API — for offline, private, zero-token-cost development.

  • What Ollama is and how it serves models on localhost
  • Pull, run and customise a model with a Modelfile
  • Switch between local and hosted models with config only
  • What fits on a 16 GB laptop — and why you still eval on the hosted model
Start →
REQUESTS IN FLIGHT (ILLUSTRATIVE)00:001506:004008:0012009:0018012:0014015:0011018:006021:0025PODS = MAX(3, ⌈LOAD ÷ 28⌉)00:003 (min)06:00308:00509:00712:00515:00418:00321:003Average ≈ 4 pods vs 7 if sized for peak → ~40% fewer pod-hours
12intermediate·11 min·9 steps

Autoscaling

Add and remove pods, workers and GPU nodes automatically from the right saturation signal — fast up, slow down, within minimum and maximum limits.

  • The levels of autoscaling: pods, events, nodes, vertical, serverless
  • Pick the saturation signal per workload — not CPU for LLM APIs
  • Targets, minimums, maximums and asymmetric scale-up/down
  • Size a day: peak 180 in flight → 7 pods, ~40% fewer pod-hours than fixed peak sizing
Start →
ROUND ROBIN — OPEN STREAMSPod A18Pod B4Pod C11LEAST CONNECTIONS — OPEN STREAMSPod A11Pod B11Pod C11Equal request counts ≠ equal load when a request can last 2 s or 2min.33 open streams, snapshot (illustrative)
13intermediate·11 min·9 steps

Load Balancing

Spread long, uneven LLM streams across healthy instances — and across model deployments — without hot spots, lost context or cut-off answers.

  • Why round robin piles long streams onto one instance
  • Algorithms, health checks, timeouts and draining for streaming traffic
  • Remove sticky sessions by moving state to Redis
  • Balance outbound calls across model deployments by quota, with failover
Start →
GATE 1 AFTER 30 MIN AT 5% (ILLUSTRATIVE)Escalation · baseline21%Escalation · canary29%Re-ask · baseline9%Re-ask · canary14%Errors and p95: unchanged — a health-only canary would have promotedthis.Gate 1 ✗ → automaticabort to 0%
14intermediate·11 min·10 steps

Canary Releases

Send a small, consistent slice of real users to a new version, compare health and answer quality against the current one, and promote in steps — or abort in seconds.

  • Why LLM regressions need quality metrics, not just errors
  • Route a consistent slice of users by hash, not per request
  • The step plan: 5% → 25% → 50% → 100%, with automatic abort
  • Read a failed canary: escalation 21% → 29%
Start →
REQUEST CONTEXTuser E12345group: IT staffregion: UShash bucket: 37KNOWLEDGE_PROMPT_VERSION — RULES IN ORDER1 · group = IT staff → v42 ✓ match2 · region = EU → v41 (not yet approved)3 · hash bucket < 10 → v424 · default → v41Result: v42logged on the traceKill switchwrite_tools_enabled = off
15intermediate·11 min·10 steps

Feature Flags

Choose code paths, prompt versions and models per request with runtime rules — to release, target, experiment and switch things off without redeploying.

  • Separate deploy from release with runtime rules
  • The four kinds of flag, including kill switches
  • How a flag evaluates rules for one request
  • Roll a prompt out from IT staff to 100% — and hold back one region without a deploy
Start →
resolvelogs versionv39v40v41v42Prompt registryknowledge_agent.systemlabel: production→ v41label: staging→ v42rag-api (prod)Traceprompt = v41
16intermediate·10 min·10 steps

Prompt Versioning

Treat prompts like code: immutable versions with author, reason and eval results, labels that point at what's live, and the version on every trace.

  • Why an unversioned prompt edit is an invisible deploy
  • Versions, labels and traces: the three parts of prompt versioning
  • Git, registry or hybrid
  • Find and roll back a bad prompt by moving a label
Start →
HOSTED CHAT MODELSEMBEDDING MODELSFINE-TUNED / SELF-HOSTEDDated snapshotnot an aliasAuto-upgrade offRetirement datetracked, plannedEmbed model v2Index kb-v2re-embedded corpusSwitch togetherquery model + indexWeights hashbase · data versionEval resultsStagescandidate → prod
17intermediate·11 min·9 steps

Model Versioning

Pin the exact chat model snapshot, embedding model and fine-tuned weights per environment, record them on every request, and treat any change as a gated release.

  • Why floating aliases and retirements change behaviour silently
  • The three kinds of model and how each is versioned
  • Why a new embedding model means a new index
  • Plan a forced migration over four months
Start →
real questionretrievegenerateanswer → check fact + latencySynthetic runnerevery 1 min · 2 regionsLoad balancerrag-api podKB indexModel provider
18intermediate·11 min·9 steps

Health Checks

Tell the platform which instances are started, alive and ready — and tell on-call when the whole pipeline stops giving correct answers.

  • Why an AI service can be up and still wrong
  • The four checks: startup, liveness, readiness, synthetic
  • Why liveness must never call the model provider
  • Shutdown: fail readiness first, drain, then finish streams
Start →
SERVICE HEALTH412requests/min0.4%error rate3.1 sp95 latency1.4 sTTFT p95ANSWER QUALITY6.1%thumbs-down21%escalation7.8%no-answer96%grounded (judge)COST & CAPACITY$0.012cost/turn68%tokens vs quota17%cache hit5 of 12podsDEPENDENCIES0.2%model 429/5xx140 msvector search p95820 msServiceNow p956 hindex age
19intermediate·12 min·10 steps

Alerts and Dashboards

Show health, answer quality, cost and dependencies at a glance, and alert the right person on symptoms — so the team finds problems before users report them.

  • What an AI dashboard shows beyond the four golden signals
  • SLOs and multi-window burn-rate alerts
  • Quality alerts, severity routing and runbooks
  • A practical alert set for an LLM assistant
Start →
Imagenew buildPrompt v43Re-ranked indexkb-2026-11-12Productionrelease 2026.11.12No-answer rate10:12 · baseline 8%18%
20intermediate·11 min·9 steps

Rollbacks

Return production to its last known-good image, prompt, model, index or config in seconds to minutes — because every change was built to be reversed.

  • Why a bundled release makes rollback a guessing game
  • Time to restore for each kind of change, from 5 s to hours
  • Design for reversibility, including expand-and-contract migrations
  • Restore in ~15 minutes by undoing the fastest suspect first
Start →
Next module · 16
LLM Infrastructure & FinOps