Building eval datasets that catch regressions

An eval set that only contains cases your model already passes is decoration. It runs green on every build, it feels like a safety net, and it catches nothing — because the regressions that hurt you live in exactly the cases you didn't think to include. The value of an eval suite is entirely a function of what's in it, and building that dataset well is a separate skill from running it.

This is the companion piece to how I run evals in general. That one is about grading and CI. This one is about the harder, more neglected half: sourcing the cases, writing criteria that actually discriminate, and keeping the set honest as it grows so it keeps catching things a year from now.

Source cases from what already broke

The best eval cases are ones you didn't invent. Your imagination is biased toward inputs you understand, which are the inputs the model already handles. The cases that catch regressions are the ones that surprised you, and the record of what surprised you is sitting in two places.

Production traces. Every real input your system has seen is a candidate. Mine the logs for the outputs that were wrong, thin, refused when they shouldn't have, or answered when they should have refused. If you log inputs and outputs together (you should — see building with LLMs field guide for the instrumentation), this is a filtering job, not a writing job. Pull the ugly ones and freeze them.

Support tickets and complaints. When someone writes in that "the assistant told me the wrong return window" or sends a screenshot of a mangled answer, that is a finished eval case handed to you for free. The exact input that produced the complaint becomes a permanent case with the correct behavior written next to it. Now that specific failure can never silently come back, because a red build will stop it.

The rule I hold: every production incident ends with a new eval case, not just a fix. The prompt patch closes today's bug. The case is what keeps it closed. A set built this way grows in exactly the direction reality is attacking you, which is the only direction that matters.

Invented cases have one legitimate job: covering shapes of input you expect but haven't seen yet, usually pre-launch. Write those to mirror the real distribution you anticipate, mark them as synthetic, and replace each with the genuine version the moment real traffic produces one. Synthetic cases are scaffolding, not the building.

Cover the tail and the known failure modes deliberately

Coverage is a design problem, not an accident. Before I trust a set, I sketch the space of inputs and make sure each region has representation. The rough map I use:

• The happy path, thinly. A few cases so a total collapse trips the wire. Not more — these are the cases you're least worried about.
• Every known failure mode. One case per way the system has broken or plausibly could: the empty field, the 4,000-word paste, the sarcastic customer, the prompt-injection attempt, the question in another language, the input where the correct answer is "I don't know." If you've built guardrails, every guardrail needs an eval case that would fire it, or you can't tell when it silently stops working.
• The tail. The rare, specific, weird real inputs from production that don't fit a neat category. These are individually low-frequency and collectively where most regressions hide.
• Ambiguity. Cases where the right answer is genuinely unclear. These test whether the model handles uncertainty gracefully instead of confidently inventing.

A set that's 90% happy path and 10% everything-else passes constantly and protects you from nothing. Aim for the inverse. Overweight the cases that scare you, because those are the ones a "small tweak" will quietly flip.

Write pass/fail criteria that discriminate

A criterion has to be checkable by a stranger — someone who wasn't in your head when you wrote the case. "Is this a good support reply?" fails that test. Broken into checkable claims, it passes: addresses the actual question asked, cites the correct policy, stays under 100 words, invents no facts, keeps a professional tone. Now each is independently pass or fail, and the score is how many hold.

Write the criteria before you look at the model's output. If you grade against whatever came back, you'll rationalize a mediocre answer into a pass because it's in front of you and it seems fine. Writing the standard first — while you're thinking about the input, not the output — is what keeps the grade honest.

The other discipline: criteria should be about the answer, not about matching your phrasing. "Mentions the 30-day return window" is a good criterion. "Says it exactly the way my golden output says it" is a bad one, because it fails correct answers worded differently and teaches you nothing about quality. Which brings up the golden-output question directly.

Golden outputs vs rubric vs judge

Three ways to grade, and the right choice depends on how pinned-down the correct answer is.

Golden outputs — a stored expected answer you compare against — are perfect when there's exactly one right answer: a classification label, an extracted field, a structured JSON payload, a yes/no decision. Comparison is code, it's free, it's deterministic, and it never drifts. Push as much of your output into checkable shapes as you reasonably can, precisely so more of it can be graded this way. But golden outputs are brittle for open-ended text: there are a thousand good ways to write a support reply, and pinning one exact string rejects 999 correct answers.

Rubric grading handles open-ended output that still has objective properties. You don't store the answer; you store the checklist. Length limits, required substrings, "contains no phone number," "cites the policy" — a mix of programmatic checks and human-checkable claims. This catches a surprising amount without a second model in the loop.

LLM-as-judge is for the qualities you can't check with code or a substring: faithfulness, tone, helpfulness, whether the answer actually resolved the question. You hand a second model the input, the output, and your rubric, and it returns a score with reasoning. Details of wiring one up are in the main evals guide; the point here is a dataset point — a judge is only as good as the rubric you feed it, and the rubric is part of your dataset. Vague rubric, useless judge. And verify the judge against your own hand-grades on thirty cases before trusting it at scale, because an unverified judge is just a confident stranger with a number.

Keep the set versioned, and don't overfit to it

The eval set lives in version control as plain files next to your code — JSON, YAML, CSV, whatever diffs cleanly. Every case is a committed artifact with a history, so when a criterion changes you can see who changed it and why. When production surfaces a new failure, the pull request that fixes it adds the case in the same diff. This is how the set grows in lockstep with what you've learned.

The trap on the other side is overfitting. If you iterate against the same cases long enough, you stop improving the system and start memorizing the test — pass rate climbs while real complaints don't. Two defenses. First, hold out a split you don't look at while iterating: score it only right before you ship, as an honest read on whether your gains generalize. Second, keep sourcing fresh cases from production, so the set can't go stale enough to memorize. When you catch your eval score improving but your users aren't happier, that gap is the tell — the fix is new cases, not more tuning against the old ones.

Wire it in so regressions can't ship

A dataset that only runs when you remember to run it will, eventually, not run on the change that breaks things. The point of building it is to put it in the path of every change: a fast, cheap subset gating each pull request, the full suite nightly, and a hard rule that a prompt or model change doesn't merge until it's green. The CI mechanics are in the main evals guide; the dataset is what makes the gate mean something.

Building the set is the tedious part — mining traces, writing criteria, resisting the urge to pad it with softballs. It's also the part that pays. A prompt change or a model swap or a new tool wired into an agent will eventually break a case you'd long forgotten, and the only thing standing between that break and your users is whether the case was in the set. Build the set well and it will be.

FAQ

Where do the cases in a good eval set come from? Real production traces and support tickets first, not your imagination. Mine your logs for inputs that produced bad or ambiguous outputs, pull the screenshots people send when something looks wrong, and turn each real failure into a permanent case. Invented cases only fill coverage gaps you can't reach from real data yet, and you replace them the moment real traffic gives you the genuine version.

What makes a pass/fail criterion good instead of vague? It has to be checkable by someone who wasn't there when you wrote it. "Is this a good answer?" is not a criterion; "cites the correct refund window, stays under 80 words, invents no policy" is three. Break every fuzzy quality into specific, independently-checkable claims, and write them down before you look at the output.

Golden outputs or an LLM judge — which should I use? Golden outputs when there's one right answer to compare against (labels, extracted fields, structured JSON). A rubric or judge when the output is open-ended and correctness is about qualities like faithfulness or tone. Most real sets mix both: pin down everything you can with exact checks, reach for a judge only for the genuinely fuzzy part.

How do I keep from overfitting to my own eval set? Hold out cases you never look at while iterating, and grow the set continuously so it can't go stale. If your pass rate climbs while real-world complaints don't drop, you're tuning to the test, not the task, and the fix is fresh cases from production.

How big does an eval set need to be to catch regressions? Coverage matters more than count. Twenty to fifty well-chosen cases that hit every distinct failure mode and input shape will catch more than a thousand near-duplicate happy-path cases. Size up by adding categories you don't yet cover, not by cloning cases you already pass.