Evaluation & Observabilityintermediate12 min · 8 steps
Benchmarks and Leaderboards
Read a benchmark score correctly: what it measures, whether it's saturated or contaminated, and which harness produced it.
heavy workresponsedata
ProblemStep 1 of 8
“88% on SWE-bench” tells you almost nothing
A headline score hides how it was produced. The same weights can score very differently depending on the agent scaffold, tool access, effort level and output-token cap, and since mid-2026 also the benchmark version, the metric, who ran it and whether a fallback model handled some tasks.
Vendor self-reports also run higher than independent leaderboards, because labs report the best configuration they found.
Watch outThe 2026 web is full of fabricated leaderboard pages. If a frontier score names a model you can't find in a primary source, assume it's SEO spam.
Lesson 3 of 5 in Evaluation & Observability