Evals That Catch Regressions

You do not need a research-grade benchmark. You need fifty cases that fail when your system gets worse, and a habit of running them before anything ships.

Start with the bugs you already had

The best eval set is a graveyard. Every time something goes wrong in production — a wrong answer, a hallucinated citation, a tool called with nonsense — turn it into a case with the correct output beside it. Within a couple of months you have a set that encodes your system’s actual failure modes rather than an imagined distribution.

Fifty real cases beat five thousand synthetic ones, because synthetic cases test the behaviour you thought to ask about.

Grade what you can grade cheaply

Not everything needs a model judge. Exact match, JSON schema validity, “does the cited passage contain the claim”, latency and cost are all deterministic and free. Reserve model-graded rubrics for genuinely subjective qualities like tone, and when you use one, pin the judge model — a judge that drifts turns your eval into noise.

Measure the pipeline stage, not the vibe

“Answer quality dropped” is not actionable. Recall@k, groundedness rate, tool-call validity and refusal correctness each point at a specific stage. When a number moves you want to know which component to open.

Run it on every change, including the ones that feel safe

Prompt tweaks, retrieval parameter changes, library upgrades and model version bumps all move quality, often in opposite directions. The value of an eval suite is entirely in being run automatically; a suite executed when someone remembers is a suite that finds nothing.

Accept a regression budget

Demanding zero regressions blocks every upgrade, because improvements are rarely uniform. Decide in advance which cases are non-negotiable — safety refusals, factual grounding — and allow movement elsewhere. This converts an argument into a decision.

What to take away

Curate from real failures, grade deterministically where you can, break scores down by stage, run automatically, and define what you will not trade away. The point is not a number that goes up. It is a number that goes down loudly when something breaks.