Skip to content
anythingai.space
LLM Infrastructure & FinOpsintermediate12 min · 9 steps

CI/CD for LLM Apps

Ship prompt, model and config changes through staged evals, a quality gate, canaries and automatic rollback.

deployNothing here measured answer qualityCommitprompt editUnit testsCI greenProductionquality dropping
requesterrorheavy workdataresponsesignal
1 / 9
ProblemStep 1 of 9

The unit tests pass. The answers got worse.

A one-line prompt edit sails through the usual pipeline: the code compiles, the template renders, CI is green. In production, answers quietly get less accurate.

LLM outputs are non-deterministic, evaluations are slow and costly, and a change can be prompt, model or data rather than code. Binary pass/fail tests can't see a drop in quality.

Traditional CI/CDLLM CI/CD
Binary pass/fail testsProbabilistic evaluation
Fast testsSlow, expensive evals
Deterministic outputsNon-deterministic outputs
Code changes onlyPrompt + model + data changes
Watch out“Set temperature to 0 for reproducible CI” no longer works: Claude rejects sampling parameters under API version 2026-10-21. Assert on structured fields and semantics, not exact strings.

Lesson 2 of 4 in LLM Infrastructure & FinOps