LLM Infrastructure & FinOpsintermediate12 min · 9 steps
CI/CD for LLM Apps
Ship prompt, model and config changes through staged evals, a quality gate, canaries and automatic rollback.
requesterrorheavy workdataresponsesignal
ProblemStep 1 of 9
The unit tests pass. The answers got worse.
A one-line prompt edit sails through the usual pipeline: the code compiles, the template renders, CI is green. In production, answers quietly get less accurate.
LLM outputs are non-deterministic, evaluations are slow and costly, and a change can be prompt, model or data rather than code. Binary pass/fail tests can't see a drop in quality.
| Traditional CI/CD | LLM CI/CD |
|---|---|
| Binary pass/fail tests | Probabilistic evaluation |
| Fast tests | Slow, expensive evals |
| Deterministic outputs | Non-deterministic outputs |
| Code changes only | Prompt + model + data changes |
Watch out“Set temperature to 0 for reproducible CI” no longer works: Claude rejects sampling parameters under API version 2026-10-21. Assert on structured fields and semantics, not exact strings.
Lesson 2 of 4 in LLM Infrastructure & FinOps