Rafael Gorski

Eval-driven development: from production traces to the release gate

GN-0003 Draft · · Trail log · Sources: Evals for AI Engineers (forthcoming), OpenTelemetry GenAI semantic conventions

An LLM system has no specification to test against, so its limits stay hidden until someone measures them.

Eval-driven development is test-driven development for LLM systems: each failure found in production becomes an eval that every later change must pass. This note splits the practice into two waves, with engineers between them: evals that watch production, and the same evals as a gate on every change. The two waves are this note's own framing. For the method in depth, see Shreya Shankar and Hamel Husain's forthcoming Evals for AI Engineers. The note ends with eight requirements a team can adopt.

  1. Wave 1: watch production

    1. Telemetry captures production traces
    2. Online evals score a sample
    3. Drops and regressions show as failures
  2. Events go to engineers

  3. Close the loop

    1. Each failure becomes a failing eval
    2. Engineers fix the system until it passes
    3. Engineers curate the eval set
  4. The eval joins the suite

  5. Wave 2: gate every change

    1. CI/CD runs the suite on every change
    2. Each module is scored against its baseline
    Pass
    the change ships
    Hold
    a person decides
    Roll back
    the module reverts

Passing changes go to production

Exhibit 1. Tinted boxes run automatically; the open box is work for engineers. The middle box is the test-driven cycle.

1Evals make the limits explicit

A conventional service can be checked against its specification. An LLM system mostly cannot. Its behaviour emerges from a prompt, a model, retrieved documents, tools and the code that joins them, and any of those can change without a line of the team's own code changing. Until that behaviour is measured, its limits are known only from demos, which show the best case, and from complaints, which arrive late.

An eval is a written, rerunnable claim about behaviour: on this set of cases, graded this way, the system succeeds this often. Each eval that fails states a limit explicitly. “The assistant is sometimes wrong about policy” becomes, for example, “retrieval answers from a superseded policy in 7% of policy questions, and the rate rose after the last index rebuild”. A limit stated that way has an owner, a size and a trend, and it can be compared before and after a change.

Public benchmarks do not do this job. They measure someone else's tasks, usually once, which is why the radar puts benchmark scores as launch evidence on Hold. Strong test scores are not enough on their own either: models with equally strong scores can behave very differently once deployed [UNDERSPEC]. The evals that make a system's limits explicit are built from that system's own behaviour.

2Eval-driven development, step by step

Test-driven development works in a short cycle: write a test that fails because the behaviour is missing, change the code until it passes, then keep the test so that every later change has to pass it too. Eval-driven development (EDD) applies the same cycle to an LLM system. A failure seen in a real trace becomes a failing eval. The system changes until the eval passes without lowering any other result. The eval then joins the suite that guards every later change.

The stepIn test-driven and in eval-driven development
Write a check that fails

TDD writes a test, from the specification, for behaviour the code does not have yet.

EDD builds an eval from real traces in which the system failed, found through telemetry.
Make it pass

TDD changes the code until the test passes.

EDD changes the prompt, the retrieval, the model, a tool or the code until the eval meets its target, with every other eval still within its tolerance.
Decide what passing means

TDD asks that an assertion holds, on every run.

EDD asks for a pass rate over a set of cases, across repeated runs, compared with a recorded baseline.
Grade the result

TDD grades with the test code.

EDD grades with code checks where the answer can be checked mechanically, and elsewhere with a model judge whose agreement with human grades has been measured.
Keep it

TDD adds the test to the suite that runs on every change.

EDD adds the eval to the suite that gates every change in CI and scores a sample of production traffic.

The analogy holds for the cycle and breaks in two places, and both matter. A passing eval is a rate, not a certainty: the same input can produce different outputs, so results are compared across repeated runs and against a baseline. And the grader can be wrong, so a failing eval has three possible causes, much as a failing property did in GN-0002: the system, the eval's cases or the grader.

The two practices fit together. GN-0002 argues for properties that state what must never happen in the deterministic code around the model, and leaves the model's own answers to evaluation (section 10 of GN-0002). EDD covers that part. Never-rules stay properties; answer quality, grounding and task success become evals.

3Wave 1: watch behaviour through telemetry

The first wave changes no release process. It measures what the system already does in production, so it starts from the traces described in GN-0001: one connected tree per request, with every model call, tool call and agent step named in the OpenTelemetry GenAI conventions [OTEL-GENAI].

Online evals score a sample of those traces as they arrive. Some are code checks: the output parses, a citation is present, a tool call succeeded. Others need judgement, such as whether an answer is supported by the documents retrieved, and use a model judge checked against human grades (section 5). The conventions define an event for the result, gen_ai.evaluation.result, which carries the evaluation's name, a score and a label, and should be parented to the span it judges [EVENTS]. Recorded that way, every score can be followed back to the trace, the module and the version that produced it.

Scored traffic is then treated like any other service indicator: tracked per module and per version, with an objective and a baseline [SLO]. The radar has answer-quality SLOs in Trial and continuous quality evaluation in production in Adopt. Two kinds of change show up:

  • Degradation is a slow drift with no release behind it: the mix of questions shifts, new topics arrive, or a provider changes something on its side. It shows as a trend against the objective.
  • Regression is a step change after a release: a new prompt, model, index or tool. Version attributes on the spans, such as gen_ai.agent.version and gen_ai.response.model, tie the step to the release that caused it [OTEL-GENAI].

Some failures appear only in production. In 2025 Anthropic described three infrastructure bugs that intermittently degraded responses, and the evaluations in place did not capture what users were reporting [QUALITY]. An offline suite cannot find a fault that exists only on the serving path. Evals that score live traffic can.

Online evals read prompts and outputs, which often carry personal data. Whether they may is the content-capture decision in section 5 of GN-0001, and the radar puts capturing prompts and outputs by default on Hold. Decide in writing which environments may be sampled, by whom, and how long the samples are kept.

4Wave 2: a guardrail for every module in CI/CD

The second wave moves the same evals into the delivery pipeline, where they act as a guardrail. Any change to a prompt, a model or its version, a retrieval index, a tool or the code is a release, and it runs the suite before it merges. The suite then runs again, on a schedule, against what is deployed: the canary first, then production. Google's SRE practice already gives launches a checklist, and compares a release with a control on a small share of traffic before it widens [LAUNCH] [CANARY]. For an LLM system, evals supply the comparison.

The guardrail works module by module. One overall score hides which part moved. A separate eval set and baseline for the router, the retrieval, the answer writer and each tool says which part to fix or roll back. Each result is read against a rule agreed before the run:

  • Pass: every eval for the module is within its allowed drop from the baseline, and no never-rule is broken.
  • Hold: a result sits at the edge of its tolerance, the evidence is too thin to decide, or a judge has not been re-checked since its criteria or the module changed. The change stops widening until a named person decides.
  • Roll back: a result falls beyond its allowed drop, or a never-rule is broken. The module returns to its baseline version, automatically where possible, and an event goes to its owner. Before a merge, rolling back simply means the change does not merge.
Module and evalBaseline, candidate and decision
Intent router: routing accuracy

400 labelled conversations. Baseline 96.0%, candidate 95.7%, allowed drop 1 point.

Pass. Within tolerance.
Retrieval: grounded answers

250 policy questions, three runs each, graded by a judge checked against human grades. Baseline 91%, candidate 84%, allowed drop 2 points.

Roll back. Seven points below the baseline. The index returns to its baseline version, and an event goes to the retrieval owner.
Answer writer: complete, on-policy answers

300 conversations, graded by a judge. Baseline 88%, candidate 86%, allowed drop 2 points.

Hold. At the edge of tolerance, and the judge has not been re-checked since the prompt changed. A reviewer grades 30 samples by hand before the change widens.
Refund tool: never-rules

10,000 generated cases for properties of the kind in GN-0002: refunds within the charge, actions within the user's permissions. Baseline 0 violations, candidate 0, allowed 0.

Pass.
Whole task: resolved conversations and cost

50 end-to-end conversations, three attempts each. Baseline pass^3 of 72% at 4.2 cents per resolved conversation; candidate 71% at 4.4 cents. Allowed: a drop of 3 points and a cost rise of 10%.

Pass.
Exhibit 2. Illustrative guardrail run for one release candidate in canary, module by module. Synthetic data: the system, its modules, the eval sets and every number are invented. Each decision follows the rules above and is written as a word.

The run in Exhibit 2 stops the release from widening. Retrieval returns to its baseline version, the answer writer waits for a person to review it, and the router and the refund tool keep the candidate. Three practices make a run like this trustworthy:

  • Tolerances come from the baseline's own noise. Run the baseline against itself several times before setting an allowed drop. If its scores vary by a point and a half between runs, a one-point drop says nothing.
  • Tasks run more than once. τ-bench reports pass^k, the chance that an agent succeeds on every one of k attempts at the same task, because a single run overstates the consistency users will see [TAU]. The radar has repeated-trial reliability in Trial.
  • The suite is versioned code. Keep evals as versioned tasks in an open harness such as Inspect, so the same suite reruns unchanged on every change and results stay comparable [INSPECT]. The radar has a versioned open evaluation harness in Trial and canary releases for model and prompt changes in Adopt.

Passing evals are evidence for the gate, not its only key. In 2025 OpenAI rolled back a model update that had shipped on positive offline evaluations and A/B results, although expert testers had said it felt off, and committed to treating such behaviour issues as launch-blocking [SYCOPHANCY]. The people who read the outputs need the authority to hold a release, not only the numbers.

5Close the loop: from eval results back to engineers

The two waves improve the system only if their results reach someone who can act on them. Each degradation flagged in production and each failed gate raises an event to the module's owner, carrying the eval's name, the scores against the baseline, the versions involved and links to the traces. The owner then curates two things, the system and the eval set, and the change goes back through the gate.

Curation starts with error analysis: reading the flagged traces, naming the ways the system failed, counting each one and fixing the largest first. Each named failure becomes an eval written from the real traces, which is the failing step of section 2. Curating the eval set also means:

  • Re-baselining on purpose. When a change is meant to move a score, the owner records a new baseline and the reason for it, so that a deliberate change is not later read as drift.
  • Refreshing the cases. Production traffic changes. Sample new traces into the eval set so that it keeps resembling what users ask, and keep the old failures as regression checks.
  • Keeping the graders honest. People refine their criteria as they grade outputs, so a judge written last quarter may encode criteria the team no longer holds [VALIDATORS]. Model judges also show position, verbosity and self-enhancement biases [JUDGE]. The radar puts LLM judges without human calibration on Hold for these reasons.
  • Writing up incidents. A regression that reached users gets a blameless review that ends in owned fixes and new evals, as the radar's blameless postmortems for AI incidents recommends.

Shreya Shankar and Hamel Husain's Evals for AI Engineers is announced to cover this ground in depth: finding and prioritising failures through error analysis and representative test sets, building evaluations from code-based checks, LLM judges, human feedback and synthetic data, and monitoring production for regressions [EVALS]. Shankar is also the first author of the study on drifting criteria cited above [VALIDATORS].

The book is forthcoming from O'Reilly in November 2026. This note does not quote it or attribute claims to it beyond the publisher's description. The two waves and the loop between them are this note's framing.

Exhibit 3 follows one regression once round the loop, from the release that caused it to the eval that now guards against it.

  1. Index rebuild passes the CI gate and ships0.5 days
  2. Online evals flag fewer grounded answersevent raised to the retrieval owner1 day
  3. Guardrail rolls the index back to its baseline0.25 days
  4. Wait for triage0.75 days
  5. Error analysis on 60 flagged traces1 day
  6. Write a failing eval from 12 real tracesfails on the shipped index0.5 days
  7. Check the judge against 50 human grades0.5 days
  8. Fix: drop superseded policies when indexing1 day
  9. Eval passes and joins the CI suite0.25 days
  10. Wait for the release window2 days
  11. Canary, then full rollout1.25 days
Exhibit 3. Illustrative turn of the eval loop for one regression, from the release that caused it to the eval that now guards against it, over nine working days. Synthetic data: the system, the failure, the counts and every duration are invented. A filled bar is work, an outlined bar is time spent waiting in a queue, and red marks the regression and the failing eval written from it.

Three points stand out:

  • The gate passed because the suite had never seen the failure. No question in the eval set had an answer that changed between policy versions. The limit was in the evals as much as in the system, and only production traffic revealed it.
  • Wave 1 caught it within a day, and wave 2 contained it to one module. Because the index is versioned on its own, the guardrail rolled back the index alone. The router, the prompt and the model stayed on the release.
  • The failure now lives in the suite. Twelve real traces became a failing eval before the fix. Once it passed, it joined the suite, so the same regression fails the gate if it returns. As on the value stream map in GN-0002, much of the elapsed time is waiting rather than work.

6Requirements for leadership

These are written so that they can be copied into an engineering standard or a release policy. Each one links to the section above that motivates it.

The key words MUST, MUST NOT and SHOULD are to be read as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.

  1. R1

    Every module of an LLM system, including each prompt, model, retrieval index and tool, MUST have a named owner, its own eval set and a recorded baseline, versioned with the module.

    §4

  2. R2

    Online evals MUST score a sample of production traces continuously. Each result SHOULD be recorded as an evaluation event on the span it judges, and a degradation or regression against the baseline MUST raise an event to the module's owner with links to the traces.

    §3 · §5 · [EVENTS]

  3. R3

    Every change to a prompt, model, retrieval index, tool or code MUST run the eval suite in CI against the baseline before it merges, and the suite MUST run again on a schedule against the canary and production. The rules for pass, hold and roll back MUST be agreed before the run.

    §4 · [CANARY]

  4. R4

    Allowed drops SHOULD be set from repeated runs of the baseline against itself, and end-to-end tasks SHOULD be run several times, with pass^k reported beside single-run success.

    §4 · [TAU]

  5. R5

    Modules MUST be versioned and deployable separately, so that a failed eval can roll back one module without reverting the whole release. That rollback path MUST be tested before launch.

    §4 · §5 · [LAUNCH]

  6. R6

    Every regression found in production MUST become a failing eval, built from the real traces, before its fix merges. Once the eval passes, it MUST join the suite.

    §2 · §5 · [QUALITY]

  7. R7

    A model judge MUST NOT gate a release until its agreement with human grades on a sample has been measured. That agreement MUST be measured again whenever the judge's criteria, prompt or model changes.

    §5 · [JUDGE] · [VALIDATORS]

  8. R8

    Named reviewers SHOULD have the authority to hold a release on qualitative findings, even when every eval passes.

    §4 · [SYCOPHANCY]

7Where to start

  1. Read the traces, then write the first evals. Pick one workflow that already has traces and a business owner. Read a hundred or so production traces with that owner, name and count the failures, write evals for the largest few, and record a baseline for each module they touch (R1, R6).

  2. Gate the next change. Run the suite in CI on every change to that workflow. Run the baseline against itself several times to set the allowed drops, write down the rules for pass, hold and roll back, and check that each module can be rolled back on its own (R3, R4, R5).

  3. Close the loop in production. Score a sample of live traces with the same evals, under a written content-capture policy. Send degradations to the module's owner as events, check every judge against human grades, and add each new failure to the suite. Then write R1 to R8 into the release policy (R2, R7, R8).

8What is still open

Being specific about the limits is part of the guidance:

  • A suite catches only failures someone has already seen. The gate in Exhibit 3 passed for exactly that reason. Online evals shorten the time it takes to see a new failure; they do not remove it.
  • Model judges are models. They carry documented biases, and checking them against people costs human time every time the criteria change [JUDGE] [VALIDATORS].
  • On small eval sets, small differences are noise. Tolerances set from repeated baseline runs reduce false alarms, but choosing them is still a judgement.
  • Passing is not proof. Models with equally strong scores can behave differently in deployment [UNDERSPEC], and a model update that passed its offline evaluations has already been rolled back [SYCOPHANCY].
  • Evals cost money. Running the suite on every change, several times over, and judging production samples with a model are workloads with a budget, like any other.
  • The OpenTelemetry evaluation event is in Development status, and events are not yet available in every language SDK [EVENTS].
  • The book this note points to is not yet published. The note will be checked against it once it is.

None of this argues for fewer tests, canaries or human reviews. It argues for giving each of them a written, rerunnable record of what the system is known to get wrong.

9References