Evaluation

You cannot improve what you cannot measure, and LLM systems are unusually easy to fool yourself about — a change that looks great on three hand-picked examples can quietly regress on the twenty you didn't check. Evaluation is the discipline of measuring quality systematically so that "it feels better" becomes "recall went from 0.71 to 0.83." It is the difference between engineering an LLM system and tinkering with one.

Documents in this section

• LLM Evaluation Basics — the vocabulary and the core idea of a test set for probabilistic systems.
• LLM-as-Judge — using a model to grade outputs, and how to do it without fooling yourself.
• Offline Evaluation Harnesses — building a repeatable eval you can run on every change.
• Regression Gates — wiring evals into CI so quality can't drop unnoticed.

Evaluation ties the whole library together: it's how you choose a chunk size, a prompt, or an agent architecture on evidence instead of vibes.