Skip to content
Episode 125 · Every change proves itself

Eval-driven development: shipping AI changes like a disciplined team

Monday, 9:00: three regressions reached guests last month, all from "small" prompt changes nobody ran the eval on. Five engineers ship about 25 AI changes a week. Lina hands you the release process: "You lead this one."

Turn every AI bug into a test case, check every change against all of them, and stop guessing whether your AI got better.

30 min advancedEval-driven development loopGolden datasets from real trafficEval regression gates in CIRegression evals in CIHill-climbing and overfitting the eval setOnline quality signals and production sampling
  1. Problem

    Five engineers change prompts, retrieval settings and tools on the hotel assistant every week. Each change fixes something; some changes silently break something else.

  2. Why the simple system fails

    An eval that someone runs by hand 'when they remember' is skipped under deadline pressure. A fixed set written once goes stale as traffic changes. Teams tune prompts until the eval score goes up and are surprised when production doesn't.

  3. Engineering concept

    Make evals the development loop itself: write the failing case first, iterate until it passes without breaking others, gate merges automatically, confirm online, and feed production failures back into the set.

  4. Technology

    Golden sets sampled and stratified from traces, an eval runner in CI with tiered suites, regression gates on protected metrics with noise allowances, held-out sets, online quality signals and sampled grading.

  5. Trade-offs

    Bigger suites catch more but slow every PR and cost money; strict gates block regressions but also block useful changes on noise; tuning on the eval improves it but risks overfitting.

01

The problem

2 min
Monday, 9:00. The monthly quality review.
  1. Omar · On-call SRE
    3 regressions reached guests last month. The worst: pet-fee answers wrong for 9 days.
  2. Karim · CTO
    We have a golden set and a calibrated judge. Why didn't they catch it?
  3. Maya · Product manager
    People run the eval "for big changes". All three were one-line prompt edits.
  4. Lina · Senior engineer, your mentor
    You own the release process now. You lead.
02

The big idea

4 min
The eval is the development loop
The big idea

Make the eval the path every change walks, not a step people remember.

A handbook rule ("run the eval on every change") is a rule people forget, which is the problem you already have. Like test-driven development, but the tests are statistical, so the gate must tell a real regression from noise.

03

How it works

5 min

uses evals as the development loop:

The loop
  1. 1
    Failing case first

    A bug becomes a golden-set case that fails today ("pet fee, Hotel Ribeira: 40 EUR"). It's the finish line, and it guards forever.

  2. 2
    Iterate on dev only

    Change prompts, retrieval or tools until it passes without breaking others (). A held-out set () of 150 cases nobody tunes on runs nightly to check gains are real.

  3. 3
    Smoke on every push

    50 stratified cases, ~4 minutes. Catches catastrophic breaks fast.

  4. 4
    Full suite gates the merge

    A regression gate (): 600 cases, protected slices run 3×, paired against main. Block if broken − fixed > 2, or McNemar's check (broken − fixed)² / (broken + fixed) > 3.84. PII leaks: zero tolerance.

  5. 5
    Canary, then everyone

    A canary releases to 5% of guests for 24 h: escalations, thumbs-down, sampled judge grades. Then 100%.

  6. 6
    Failures become cases

    Weekly triage turns every escalation and complaint into a new labelled case.

04

Watch a decision being made

4 min
Worked example

You build the release loop; Lina checks

No prompt, retrieval, tool or model change reaches all guests without proving it doesn't regress what matters.

  1. 1Start from a failing case

    "Do you allow dogs at Hotel Ribeira?" expecting 40 EUR, plus two hotels with other fees. They fail on main today.

    Why: A case that fails today is the bug written down; passing proves the fix.

  2. 2Write the gate before the numbers

    Protected slices, allowances and zero-tolerance metrics in config, before any run.

    ts
    const mcnemar = (broken - fixed) ** 2 / (broken + fixed);
if ((broken > fixed && mcnemar > 3.84) || broken - fixed > maxNetBroken) return "block";

    Why: Rules written after seeing results invite motivated reasoning; noise-aware rules keep people from ignoring the gate.

  3. 3Test it on a real PR

    "Shorter answers": refund slice 12 broken, 4 fixed. (12 − 4)² / 16 = 4.0 > 3.84, and net broken 8 > 2. Blocked.

    Why: Both rules fire: significant, and over the allowance.

  4. 4Confirm online

    After merge, 5% canary for 24 h, automatic rollback if escalations rise. Lina: "Yes."

    Why: Offline says "not worse on what we know"; only real guests say "better".

The whole thing in one line: Failing case first, tiered suites, a gate written in advance, an untouched held-out set, a canary: all three of last month's regressions would have been blocked.
05

In real life

3 min
Prove it before merge, or prove it live?

Google: automated tests before code lands

  • Widely described as running large test suites on changes before submission
  • Catches breakage early, when it's cheap
  • Can't tell whether users will like it

Strong at "not worse on what we know".

Netflix: online experiments

  • Publicly known for A/B testing product changes on real members
  • Measures what users actually do
  • Some users see the worse version

Strong at "better for real users".

06

What goes wrong

3 min
Running evals by hand for "big" changes only

Most regressions come from small edits.

Instead

Smoke on every AI push, full suite gating every merge, automatically.

Gates that block on any drop

Noise fails PRs; people re-run until green or disable the gate.

Instead

Paired, noise-aware rules; zero tolerance only for safety and privacy.

A golden set written once

Traffic drifts; the set measures last year's guests.

Instead

Weekly failure triage, monthly refresh per slice.

07

Trade-offs

3 min
TierChoose forYou pay
Smoke suite (50)Every push, first tierToo small to see 2–5 point drops
Full suite, repeated runsEvery merge of AI changes~25 min and real tokens per PR
Strict zero-regressionPII, safety, critical tool actions onlyBlocks on noise elsewhere
Online canaryAfter offline gates passSome guests see the worse version
Wednesday, 11:00. Your gate blocked a PR.
  1. Omar · On-call SRE
    Block anything that breaks a protected case. One broken refund case is one wrong answer.
  2. Karim · CTO
    Yesterday's PR broke 2 refund cases and fixed 3. That rule blocks an improvement. Re-running gave different flips.
  3. Omar · On-call SRE
    So we tolerate some regressions?
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • Which changes must run the evals: every PR touching a prompt, tool description, retrieval setting or model id.
  • The tiers and their cost (smoke per push, full before merge, held-out nightly, 5% canary), and the gate rule, written before any run.
  • Who may see the held-out set (nobody tunes on it) and how production failures become cases.
Tell Claude Code
Last month 3 regressions reached guests from small AI changes nobody evaluated. Wire evals into CI: on pushes touching `prompts/`, `tools/`, `retrieval/` or model config, run the 50-case smoke suite; before merge, the 600 dev cases (protected slices 3 runs); nightly, the 150 held-out cases. Post per-slice broken and fixed cases on the PR. Block when a protected slice has broken − fixed > 2 or McNemar > 3.84 with more broken, and on any PII leak. Never tune on held-out. Plan first, then replay the pet-fee PR.
Check before you accept
  • The replayed pet-fee change fails the gate and the report names the broken cases.
  • 7 broken and 3 fixed gives (7 − 3)² / 10 = 1.6: a warning, not a block.
  • Only the nightly job reads the held-out set.
Red flags
  • Re-running a failing eval until it turns green.
  • Case-specific lines added to the prompt, or failing golden cases edited away.
  • The held-out set run during tuning "just to check".
09

Check yourself

2 min
Q1/3Reasoning
On 300 paired cases, a candidate breaks 7 cases and fixes 3. The overall pass rate drops from 86.0% to 84.7%. Using (b − f)² / (b + f) > 3.84 as the significance check, what should the gate do?
How confident are you?

Next episode: a review page told StayHub's staff assistant to email today's arrival list to a stranger, and the assistant tried.

Finished?

Mark it complete to update your skill map.

Next: AI security: assume the model will be hijacked
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Tiers, gates and a held-out set
Worked example

You build StayHub's eval-driven release loop (Lina checks)

Make it impossible for a prompt, retrieval, tool or model change to reach all guests without proving it doesn't regress what matters.

  1. 1Start from a failing case

    Turn the pet-fee incident into cases: "Do you allow dogs at Hotel Ribeira?" expecting mustMention: ["40 EUR"], plus two other hotels with different fees. Confirm they fail on main today.

    Why: A case that fails today is the bug, written down. When it passes, the fix is proven; staying in the set, it guards forever.

  2. 2Tier the suites

    Smoke: 50 stratified cases on every push (~4 min). Full: 600 dev cases before merge, protected slices run 3× (~25 min). Held-out: 150 cases nightly on main.

    Why: Fast feedback while iterating, real precision before merging, and an honest check nobody can tune against.

  3. 3Write the gate before the numbers

    Protected: refund groundedness (block if broken − fixed > 2 or McNemar > 3.84), tool-use accuracy (−2 points max), PII leaks (zero). Warn only: length +20%, cost +15%.

    ts
    const mcnemar = (broken - fixed) ** 2 / (broken + fixed);
if ((broken > fixed && mcnemar > 3.84) || broken - fixed > maxNetBroken) return "block";

    Why: Rules written after seeing results invite motivated reasoning. Noise-aware rules stop the gate blocking on chance, so people don't learn to ignore it.

  4. 4Keep a held-out set honest

    Move 150 cases to golden/holdout.jsonl. Nobody runs them while iterating. If dev rises and held-out doesn't, review the prompt diff for case-specific patches.

    Why: A week of tuning can teach the prompt the dev cases. The held-out set is the only number that can't be learned.

  5. 5Canary with online signals

    After merge, 5% of chats for 24 h. Watch escalation rate, thumbs-down, rephrased questions and judge grades on a 1% sample. Roll back automatically if escalations rise beyond the agreed limit.

    Why: Offline says "not worse on what we know". Only real guests say "better", and only they reveal what the set doesn't contain.

  6. 6Close the loop

    Weekly triage: every escalation, complaint and canary failure becomes a candidate case, anonymised, tagged, with the reason it exists.

    Why: The set keeps up with real traffic. Each incident can only happen once.

The whole thing in one line: You made evals the path: failing case first, tiered suites, a noise-aware gate written in advance, an untouched held-out set, a canary on real guests, and a weekly loop that turns failures into cases.
SignalOffline (CI)Online (production)
Correctness / groundednessGraders on the golden setJudge on a daily 1% sample
Task successExpected tool calls and outcomesBooking completed, no escalation
Dissatisfactionn/aThumbs down, rephrasing, asking for a human
SafetyRed-team and PII cases (zero tolerance)Classifier flags, staff reports
Latency / costMeasured per casep95 latency and cost per chat
Offline answers "not worse on what we know?"; online answers "better for real guests?". Neither replaces the other.
11

Interactive simulation

12

Visual diagram

Click a component

The EDD loop: production traces feed the golden set, every PR runs the eval suite in CI behind a gate, a canary confirms online, and new failures flow back.

13

Analogy

  • A careful restaurant kitchenComplaints go in the chef's bookEval-driven developmentReal failures become golden cases
  • A careful restaurant kitchenOld favourites tasted after any changeEval-driven developmentThe regression gate
  • A careful restaurant kitchenTasting several batches, not one spoonEval-driven developmentEnough cases and repeated runs
  • A careful restaurant kitchenCooking only to please one inspectorEval-driven developmentOver-tuning to the eval
  • A careful restaurant kitchenChecking what guests leave on platesEval-driven developmentOnline metrics after release

Where the analogy breaks: a kitchen tastes a handful of plates. Your suite can taste a thousand in minutes, every time anyone changes anything.

14

Why it matters

Without a loop, AI teams fall into two traps.

Whack-a-mole: a complaint arrives, someone edits the prompt, the complaint goes away, and two other behaviours quietly change. Three weeks later the first bug is back.

Paralysis: the system works, nobody knows why, and every upgrade is postponed because nobody can prove it's safe.

EDD fixes both. Every bug ever fixed becomes a case that can't sneak back. Every change, even a model upgrade, comes with a scoreboard. Teams with this loop can test a new model in a day. Teams without it take months.

15

Check your understanding

Q1/3Reasoning
Your cancellation slice has 100 cases and a margin of error of about ±7 points. Roughly how many cases do you need for ±3.5 points?
cases
How confident are you?
16

Practice ladder

Step 1 · Trace

Trace three PRs through the gate

Scenario Warm-up

Gate rules: protected refund groundedness blocks if (broken > fixed and McNemar > 3.84) or broken − fixed > 2; PII leaks: 0 allowed; cost: warn above +15%. McNemar = (broken − fixed)² / (broken + fixed).

Trace each PR: 1. Retrieval tweak: refund 3 broken, 5 fixed; PII 0; cost +4%. 2. Shorter answers: refund 12 broken, 4 fixed; PII 0; cost −20%. 3. New tool description: refund 1 broken, 1 fixed; PII 1 failure; cost +18%.

Step 2 · Complete

Finish the suites config

Code Practice

Lina's suites.yaml has two gaps: the holdout suite and the protected rule for tool-use accuracy and PII. Fill them in and say in one line why each is set that way.

suites:
  smoke:   { cases: golden/dev.jsonl, sample: 50, stratify_by: [intent] }
  full:    { cases: golden/dev.jsonl, runs_per_case: 3 }
  holdout: ___
gate:
  baseline: main
  protected:
    - { metric: groundedness, slice: intent=refund, max_net_broken: 2 }
    - ___
    - ___
  warn:
    - { metric: cost_per_case_eur, max_increase_pct: 15 }
Step 3 · Solve

Fix the breakfast-hours bug, eval first

Scenario Practice

Support reports: the concierge says breakfast ends at 10:30 at every hotel, but three hotels end at 10:00 and one at 11:00. Walk through the eval-driven fix from report to full rollout, naming each step and what you check.

Step 4 · Decide

Skip the canary for a model upgrade?

Decision Stretch

The model provider released a new version. On the full suite: no protected regressions, booking slice +3 points (significant), p95 latency −0.6 s, cost −10%. Held-out also +2 points. Karim: "Offline is clearly better. Skip the 24-hour canary and switch everyone today, we're launching in Spain tomorrow."

Write a short decision: "We chose X over Y because…, accepting the risk of…".

17

Explain it yourself

A new engineer asks why their one-line prompt fix needs a 25-minute eval run and why the gate didn't block when the score dropped by one point. Explain the loop and the reasoning.

0 words