Testing without coding: make the agent prove it
Monday, 9:12. 37 guests at New York hotels paid a cancellation fee they didn't owe, about 140 € each. The new free-cancellation rule shipped on Thursday. Claude Code's pull request had 24 tests, all green.
Claude Code writes the tests. You decide what they must check, and you make sure they can actually fail.
- Problem
Claude Code builds a cancellation-fee rule for a hotel platform and reports 24 passing tests. A week later, guests are charged fees they don't owe.
- Why the simple system fails
The agent wrote the tests from its own code, so they confirm what the code does, not what the business asked for. Nobody checked whether a single test could fail.
- Engineering concept
Tests are the spec in executable form. The human owns the examples (Given/When/Then), the agent turns them into tests that fail first, and the human checks that each test can catch a broken version.
- Technology
Example tables, acceptance and regression tests, red-then-green, break-it-on-purpose checks (mutation testing), CLAUDE.md rules that protect tests, CI that runs everything.
- Trade-offs
Thorough examples cost time up front and slow small changes; skipping them is fast until money or trust is at stake. Test depth should follow risk.
The problem
2 min- Omar · On-call SRE37 guests charged the first night, ~140 € each. All cancelled about two days before check-in. Free cancellation runs until 48 hours before.
- Maya · Product managerClaude Code's PR had 24 tests, all green. I looked.
- Lina · Senior engineer, your mentorWho decided what those 24 tests expect?
The big idea
4 minYou own the expected answers; the agent writes the test code. Then make every test prove it can fail.
A test is only as good as its expected answer. Here every answer was read from the code, so the tests said "the code does what the code does". Paris time is right for Paris hotels, and wrong by 6 hours in New York.
How it works
5 minIf the builder writes the inspection checklist after building the staircase, every line says "present: yes". You write the checklist from the building code first, and the inspector leans on the handrail to see if it gives.
- 1You write examples
An comes from the requirement, not the code. Write each as Given / When / Then: situation, action, expected result. Include a New York hotel and exactly 48 h.
- 2Agent: tests, red
One test per example, named after it. They run and fail: no feature yet. A test that passes with no feature checks nothing.
- 3Agent: feature, green
It writes the rule until the tests pass, without editing them. If one looks wrong, it stops and explains.
- 4You: break it on purpose
"Change 48 to 72, run the tests, then revert." A test must go red. This is a hand-made .
- 5Lock it in
CLAUDE.md forbids silent test edits; every bug fix starts with a that fails first; CI runs everything on every push.
You never read test code. You read names against your table, output (passed, failed, skipped) and the diff of test files.
Watch a decision being made
4 minLina rebuilds the cancellation rule with tests you own
Fix the New York bug so the rule can't silently break again, without writing test code.
- 1Write the examples
Six rows in the spec: 72 h (no fee), 47 h (fee), exactly 48 h (no fee), New York hotel at 10:00 local two days before (no fee), non-refundable rate (full fee), cancelling twice (rejected).
Why: Expectations from the business rule can disagree with the code. The New York row is the one the agent didn't imagine.
- 2Regression test first
"Add a test for the New York row and show me it fails on the current code. Don't fix anything yet."
Why: Failing first proves it catches this exact bug.
- 3Fix without touching tests, then check names
"Use the hotel's time zone. Don't change any existing test. Paste the suite summary. List every test name." Tick each name against the six rows.
Why: Naming the forbidden move closes the easiest path to green; names are how you review a suite without reading it.
- 4Break it on purpose
"Change 48 to 72, run the tests, show the output, revert." The 47 h and 72 h rows go red.
Why: If nothing goes red, the tests are decoration, however many there are.
In real life
3 minSQLite
- Publicly documents tests far larger than the code
- Runs inside billions of devices
- A bug is slow and costly to patch everywhere
Exhaustive testing is cheaper than failure.
Early Facebook ("move fast")
- Famous for shipping quickly
- Relied on fast rollback and watching metrics
- Most mistakes cheap and quickly visible
Speed was worth the occasional break.
What goes wrong
3 minThe quickest path to green is weakening tests.
"Make the code satisfy these tests; don't edit them; if one looks wrong, stop and explain."
Time zones, boundaries and special rates are where money goes wrong.
Add a non-default hotel, the exact boundary and the opposite case to every rule.
It returns the next time anyone touches the code.
Every fix starts with a failing test for that bug.
Trade-offs
3 min| Testing depth | Fits when | You pay |
|---|---|---|
| Full examples + red first + break-it | Money, personal data, bookings | About an hour per feature |
| Agent tests after, you review names | Low-stakes internal tools | Tests can mirror bugs |
| Smoke test only | Throwaway prototypes, one-off scripts | Nothing about rules or edges |
- Omar · On-call SREEvery change gets a full example table, red-first tests and a break-it check. No exceptions.
- Karim · CTOMaya's arrivals list is read-only. An hour of examples for that is waste.
- Omar · On-call SREThe fee rule looked simple too.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The examples and their expected answers (72 h, 47 h, exactly 48 h, New York), written from the spec before code exists.
- How deep to test each change: thorough for money, bookings and personal data; light for internal read-only tools.
- Whether a failing test or the code is wrong: only a human who knows the rule changes an expected value.
Implement the rule in docs/specs/free-cancellation.md. First write one test per row of the examples table, named after the row, and show me they fail. Then implement it using the hotel's time zone until they pass. Don't edit, skip or delete any test; if one looks wrong, stop and explain. Then change 48 hours to 72, show which tests fail, and revert. Paste the full suite's summary line.
- Tick each test name against your examples table; a row without a test is a gap.
- You saw red before green, and red again on the deliberate break.
- The test-file diff has no loosened assertions,
.skip, or expected values you didn't approve.
toBe(180)becomestoBeGreaterThan(0)on the way to green.- Every test uses the same default case (one city, UTC "for simplicity").
- "All tests pass" with no command or output shown.
Check yourself
2 minNext episode: the tests catch a bug, Claude Code's third fix "works", and the invoices are still wrong. Time to learn how to debug with an agent.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Rebuilding the cancellation rule with tests you own
Fix the New York bug and make sure the cancellation rule can't silently break again, without writing test code yourself.
- 1Write the examples
Six rows in
docs/specs/free-cancellation.md: 72 h before (no fee), 47 h (fee), exactly 48 h (no fee), New York hotel at 10:00 local two days before (no fee), non-refundable rate (full fee), cancelling twice (rejected).Why: The expectations come from the business rule, so they can disagree with the code. The New York row is the one the agent didn't imagine.
- 2Start with a regression test
Ask Claude Code to add a test for the New York row and run it before fixing anything. It must fail, reproducing Thursday's bug.
textAdd a regression test for row 4 of the examples table (New York hotel). Run it and show me it fails on the current code. Don't fix anything yet.Why: A that fails first proves it catches this exact bug. If it passes now, it's not testing what you think.
- 3Fix, without touching the tests
Then: "Fix the deadline to use the hotel's time zone. Do not change any existing test. Run the full suite and paste the summary."
Why: Naming the forbidden move closes the easiest path to green.
- 4Check the names against the table
"List every test name in cancellation.test.ts." Tick each against the six rows. Any row without a test? Any test with a time zone set to UTC "for simplicity"?
Why: This is how you review a test suite without reading test code.
- 5Break it on purpose
"Temporarily change 48 to 72, run the tests, show me the output, then revert." You expect the 47 h and 72 h rows red.
Why: If nothing goes red, the tests are decoration, however many there are.
- 6Lock it in
Add the testing rules to CLAUDE.md and check that CI runs the suite on the pull request.
markdown## Tests - Tests come from examples in docs/specs/; show them failing first. - Never edit, skip or delete an existing test to pass. Stop and explain. - Every bug fix adds a regression test that fails before the fix.Why: Rules written once apply to every future session; CI makes "it passes on my side" irrelevant.
| Level | Ask for it when | Example in this rule |
|---|---|---|
| A rule or calculation | Fee for a booking at a given time | |
| Code + database or a sandbox service | Cancellation saved, refund request sent to the payment sandbox | |
| A key user journey | Guest cancels from the booking page and sees the right fee | |
| After every bug | The New York row |
Visual diagram
Who owns what when an agent writes both the feature and its tests.
Analogy
- Building a staircaseYour checklist from the building codeAgent-written codeYour examples from the spec
- Building a staircaseThe builder builds to your listAgent-written codeThe agent codes until tests pass
- Building a staircaseBuilder writes the checklist afterwardsAgent-written codeTests copied from the code
- Building a staircaseInspector leans on the handrailAgent-written codeBreak the code on purpose
Where it breaks: a staircase is inspected once. Code changes every week, so your tests run again on every change, automatically.
Why it matters
Agents sound just as sure when they're wrong. Tests you own are how you keep control of behaviour you never typed.
Money hides in edge cases. The normal case almost always works. A hotel in another time zone, a deadline hit to the minute, a special rate: that's where guests get charged wrongly.
Agents chase "done". Ask an agent to make tests pass and it may change the expected answer or switch a test off. Green was the goal, so it found a way.
Tests remember the rules. Months later, someone asks Claude to change the feature. The tests are what tell it what must keep working.
Check your understanding
- 1.Write Given/When/Then examples from the spec, including edge cases
- 2.Agent breaks the code on purpose; you see a test go red; it reverts
- 3.CI runs the full suite on the pull request
- 4.Agent writes one test per example and shows them failing
- 5.Agent implements until the tests pass, without editing them
Practice ladder
Which test would have gone red?
Debugging Warm-upThursday's cancellation rule computed the deadline at 15:00 Paris time for every hotel, instead of 15:00 in the hotel's own time zone. New York is 6 hours behind Paris.
Here are four of Claude's test names, and two you could have added. For each, trace: does it pass or fail on the buggy code?
- "Paris hotel, cancel 72 h before → no fee"
- "Paris hotel, cancel 47 h before → fee"
- "Paris hotel, cancel exactly 48 h before → no fee"
- "Lyon hotel, cancel 50 h before → no fee"
- (yours) "New York hotel, cancel at 10:00 NY time, 2 days before → no fee"
- (yours) "New York hotel, cancel at 08:00 NY time, 2 days before → no fee"
Finish Lina's example table
Design PracticeLina started the examples for the free-cancellation rule (flexible rate: free until 48 h before the hotel's local check-in at 15:00, then the first night is charged; non-refundable rate: always the full stay). Fill in the three missing rows (marked ?).
| # | Given | When | Then |
|---|---|---|---|
| 1 | Paris, flexible | cancel 72 h before | no fee |
| 2 | Paris, flexible | cancel 47 h before | first night |
| 3 | ? | ? | ? (the boundary) |
| 4 | New York, flexible | cancel 10:00 NY, 2 days before | no fee |
| 5 | ? | ? | ? (the other rate) |
| 6 | ? | ? | ? (repeat action) |
Examples for the pet fee
Design PracticeMaya's next feature: "Guests can bring pets. 25 € per pet per night, maximum 2 pets. Service animals are always free and don't count toward the maximum. Hotels can turn pets off entirely."
Write 6–8 Given/When/Then examples for Claude Code, then write the one-sentence instruction that tells it how to use them.
How much testing for loyalty points?
Decision StretchKarim wants loyalty points next month: guests earn 1 point per euro spent on completed stays; points expire 12 months after they're earned; 100 points = 5 € off a future booking. Claude Code estimates two days of work. Maya also wants a read-only internal page showing each hotel's top 20 guests by points.
Decide the testing depth for each part. Write: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
A colleague says: "Claude Code writes great tests, so I just let it write the feature and the tests, and I check that CI is green." Explain what can go wrong and what you'd do instead.