Skip to content
anythingai.space
Module 21

Evaluation & Observability

LLM evaluation, LLM-as-judge, tracing and observability, and reading benchmarks honestly.

lessons
5
steps
44
time
64 min
1–5%FRONTIER JUDGE, $/DAY AT 100K TRACESOpus 5.5$1,800GPT-6 Astra$4,500JUDGE LATENCY (VENDOR-REPORTED)Luna-2152 msGPT-4o judge3,200 msTraffic100K traces/dayDistilled judge100% of tracesFrontier judge1–5% sampleHuman reviewdisagreements
01intermediate·13 min·9 steps

LLM Evaluation

Score open-ended outputs on several dimensions with code checks, LLM judges and humans, and know where each one lies to you.

  • Why one accuracy number can't grade open-ended text
  • How an LLM judge works, and the biases it brings
  • The layered judge stack: cheap inline, frontier to calibrate, humans for truth
  • Grading agents: pass^k and the trajectory, not just the answer
Start →
MEAN JUDGE SCORE PER 1,000-REQUEST WINDOW (ILLUSTRATIVE)baseline4.1this week3.6 · −12%Drop > 10% from baseline → warning alertLLM appSampler5% of requestsLLM judgeasyncDrift checkvs baseline−12%
02intermediate·12 min·9 steps

LLM Observability

Logs, metrics and traces for systems where a fast, available service can still be failing because its answers are bad.

  • Why uptime and latency aren't enough for an LLM system
  • Read a trace as nested spans across a RAG pipeline
  • Sample quality with a judge and alert on drift
  • Catch instrumentation that goes silent after an upgrade
Start →
testpublish bestVARIANTS SCORED PRIVATELY, ONE PUBLISHEDv1v2v3v4v5v6…v19…v26v27Read the confidence intervals: gaps under~15–20 Elo are noiseModel providerPrivate variantse.g. 27 tested27×Public Arenabest variant only
03intermediate·12 min·8 steps

Benchmarks and Leaderboards

Read a benchmark score correctly: what it measures, whether it's saturated or contaminated, and which harness produced it.

  • Why a bare benchmark percentage is uninterpretable
  • Spot saturation and contamination
  • Pin the harness, effort, metric and runner before comparing
  • Use Arena Elo and composite indices responsibly
Start →
DIETARY RESTRICTIONveganvegetariangluten-freeketononeCUISINEItalianAsianMexicanMediterraneanAmericanMEAL TYPEbreakfastlunchdinnersnackdessertSKILL LEVELbeginnerintermediateadvanced5 × 5 × 5 × 3 = 375 combinations → sample~25 tuples
04beginner·14 min·9 steps

Evals in Practice

Go from vibe checks to trustworthy numbers: read traces, name the failure modes, then build and validate a judge for each one.

  • Generate diverse test inputs with dimensional sampling
  • Run error analysis: open coding, axial coding, counting
  • Know when to stop reading traces
  • Validate a judge with TPR and TNR instead of agreement
Start →
EVAL PLATFORMspanscuratescoreYour AI appinstrumentedTracesspans per stepDatasetscurated from tracesEvaluatorscode + LLM judgeExperimentsrun over runPromptsversioned
05intermediate·13 min·9 steps

Eval Tooling: LangWatch and Langfuse

What an eval platform gives you (traces, datasets, experiments, online evaluators) and how to keep judge costs and latency in check.

  • What a trace must capture before any eval is possible
  • Run experiments on datasets and compare them run over run
  • Find the failing stage in a multi-step pipeline
  • Keep online checks fast and offline judges cheap
Start →
Read deeper

Guide chapters · Evaluation & Observability

Next module · 22
Design Patterns