Deployment & Delivery
Docker, Kubernetes, CI/CD, eval gates, IaC, GPUs, autoscaling, canaries, flags, versioning, health checks and rollbacks.
- lessons
- 20
- steps
- 186
- time
- 221 min
Docker
Package the app and its exact runtime into one immutable image that runs the same on a laptop, in CI and in production.
- Why fragile LLM stacks need an immutable image
- Dockerfile, image, container and registry — and how layers are cached
- Cut a RAG service image from ~1.6 GB to ~350 MB and rebuilds from 4 min to 15 s
- The mistakes that leak secrets or cut off streams
Docker Compose
Describe the API, worker, database, cache, local model and tracing in one YAML file and start the whole stack with one command.
- Why hand-installing a RAG stack costs a day and drifts
- Services, networks, volumes and profiles on one private network
- Health checks and depends_on so services start in the right order
- Where Compose stops and an orchestrator takes over
Kubernetes Basics
Declare what should run — image, replicas, resources, how to reach it — and let the orchestrator schedule, restart, scale and roll it out.
- Desired state and the reconciliation loop
- The core objects: Pod, Deployment, Service, Ingress, ConfigMap/Secret, HPA, node pools
- How a rolling update and probes keep traffic on healthy pods
- Size the rag-api: 200 concurrent ÷ 40 per pod 5 pods + headroom
CI/CD Pipelines
Verify every change — including prompts, models and retrieval settings — and release it in safe, gradual, reversible steps.
- Why LLM systems change through prompts, models and data — not just code
- The CI stages for a pull request, including fast retrieval checks and an eval gate
- CD: same image → staging → canary → 100%, with automatic rollback and a release record
- Read the DORA metrics before and after
GitHub Actions
Run tests, image builds and parallel eval suites as YAML workflows in the repo — fast, cheap per run and locked down.
- Workflows, events, jobs, steps and runners
- An LLM pipeline: test → build → eval matrix → report → deploy-staging
- Short-lived cloud credentials with OIDC instead of stored keys
- Cut a 30-minute serial eval to ~5 minutes with a matrix
Eval Gates in CI
Turn eval suites into required checkpoints that block a change — code, prompt, model or index — from advancing when quality, safety, cost or latency slip.
- Why an eval nobody must pass is advice, not a control
- The three gates: pre-merge, pre-production and canary
- Write gate policies as code with baselines, thresholds and slices
- Follow a cheaper model through all three gates — and see where it gets blocked
Environment Configs
Keep what differs between dev, staging and production out of the code and the image, so one tested image behaves correctly everywhere.
- How a silent default sent staging tests to production
- Classify every setting: secret, environment config, flag or constant
- What differs between dev, staging and production for an LLM assistant
- Typed, validated config with clear precedence
Infrastructure as Code
Define model deployments, search indexes, clusters, vaults and permissions in versioned files that are planned, reviewed and applied by a pipeline.
- Why portal clicking doesn't scale: drift, no review, no rebuild
- The IaC loop: declare → PR → plan + policy → merge → pipeline apply → drift check
- Modules per environment and remote, locked state
- Add a second-region model deployment in under an hour
Serverless Functions
Run small pieces of code on events — uploads, queue messages, schedules, webhooks — that scale from zero and bill per execution, and know where they don't fit.
- Triggers, instances that scale from zero, and per-execution billing
- Cold starts, time limits, retries and why handlers must be idempotent
- Good and poor fits for LLM systems
- Cost out ingestion vs a chat API on functions
GPU Instances
Choose GPU VMs or nodes for self-hosted models by memory, bandwidth, interconnect, availability and price — and size them for weights plus KV cache.
- Why sizing by weights alone ends in out-of-memory errors
- What matters in a GPU: memory, bandwidth, precision, interconnect, price
- Weights parameters × bytes per parameter, plus 20–40% for KV cache
- Fit a 70B model: BF16 vs FP8 vs 4-bit
Ollama for Local Models
Run open-weight chat and embedding models on your own machine behind an OpenAI-compatible API — for offline, private, zero-token-cost development.
- What Ollama is and how it serves models on localhost
- Pull, run and customise a model with a Modelfile
- Switch between local and hosted models with config only
- What fits on a 16 GB laptop — and why you still eval on the hosted model
Autoscaling
Add and remove pods, workers and GPU nodes automatically from the right saturation signal — fast up, slow down, within minimum and maximum limits.
- The levels of autoscaling: pods, events, nodes, vertical, serverless
- Pick the saturation signal per workload — not CPU for LLM APIs
- Targets, minimums, maximums and asymmetric scale-up/down
- Size a day: peak 180 in flight → 7 pods, ~40% fewer pod-hours than fixed peak sizing
Load Balancing
Spread long, uneven LLM streams across healthy instances — and across model deployments — without hot spots, lost context or cut-off answers.
- Why round robin piles long streams onto one instance
- Algorithms, health checks, timeouts and draining for streaming traffic
- Remove sticky sessions by moving state to Redis
- Balance outbound calls across model deployments by quota, with failover
Canary Releases
Send a small, consistent slice of real users to a new version, compare health and answer quality against the current one, and promote in steps — or abort in seconds.
- Why LLM regressions need quality metrics, not just errors
- Route a consistent slice of users by hash, not per request
- The step plan: 5% → 25% → 50% → 100%, with automatic abort
- Read a failed canary: escalation 21% → 29%
Feature Flags
Choose code paths, prompt versions and models per request with runtime rules — to release, target, experiment and switch things off without redeploying.
- Separate deploy from release with runtime rules
- The four kinds of flag, including kill switches
- How a flag evaluates rules for one request
- Roll a prompt out from IT staff to 100% — and hold back one region without a deploy
Prompt Versioning
Treat prompts like code: immutable versions with author, reason and eval results, labels that point at what's live, and the version on every trace.
- Why an unversioned prompt edit is an invisible deploy
- Versions, labels and traces: the three parts of prompt versioning
- Git, registry or hybrid
- Find and roll back a bad prompt by moving a label
Model Versioning
Pin the exact chat model snapshot, embedding model and fine-tuned weights per environment, record them on every request, and treat any change as a gated release.
- Why floating aliases and retirements change behaviour silently
- The three kinds of model and how each is versioned
- Why a new embedding model means a new index
- Plan a forced migration over four months
Health Checks
Tell the platform which instances are started, alive and ready — and tell on-call when the whole pipeline stops giving correct answers.
- Why an AI service can be up and still wrong
- The four checks: startup, liveness, readiness, synthetic
- Why liveness must never call the model provider
- Shutdown: fail readiness first, drain, then finish streams
Alerts and Dashboards
Show health, answer quality, cost and dependencies at a glance, and alert the right person on symptoms — so the team finds problems before users report them.
- What an AI dashboard shows beyond the four golden signals
- SLOs and multi-window burn-rate alerts
- Quality alerts, severity routing and runbooks
- A practical alert set for an LLM assistant
Rollbacks
Return production to its last known-good image, prompt, model, index or config in seconds to minutes — because every change was built to be reversed.
- Why a bundled release makes rollback a guessing game
- Time to restore for each kind of change, from 5 s to hours
- Design for reversibility, including expand-and-contract migrations
- Restore in ~15 minutes by undoing the fastest suspect first