Skip to content
anythingai.space
Evaluation & Observabilityintermediate13 min · 9 steps

LLM Evaluation

Score open-ended outputs on several dimensions with code checks, LLM judges and humans, and know where each one lies to you.

QuestionLLMResponse Acorrect, poorly writtenResponse Bfluent, incompleteResponse Ccomplete, off-topic
requestresponsedataerrorsignal
1 / 9
ProblemStep 1 of 9

There's no single right answer to compare against

Classic ML has one label per input and a clean metric like accuracy or F1. An LLM writes open-ended text, and there are many valid ways to answer the same question.

Worse, quality has several independent axes. A response can be correct but poorly written, fluent but incomplete, or complete but off-topic. One score hides which axis failed.

Traditional MLLLM systems
Single correct answerMany valid responses
Objective metricsSubjective quality
Easy to automateNeeds judgment
Static test setsDiverse scenarios
InsightMeasure each dimension on its own. “Is it good?” is several questions.

Lesson 1 of 5 in Evaluation & Observability