Debugging with AI: evidence before fixes
Wednesday, 16:40. Monthly invoices go out tomorrow, the 1st, at 06:00. For about 210 of 10,000 hotels, the PDF won't generate. Claude Code has tried three fixes. The third one works: no more errors.
Don't let the agent guess fix after fix. Make it find the cause first, with evidence.
- Problem
Monthly hotel invoices fail to generate for about 210 of 10,000 hotels, the night before they're sent. Claude Code has tried three fixes; the third makes the error disappear.
- Why the simple system fails
Asking an agent to "fix this error" invites guesses: a default value, a retry, a catch-all that hides the error. Each guess changes the symptom, not the cause, and the session fills up with wrong theories.
- Engineering concept
Debugging is a loop of evidence: reproduce, collect facts, form competing hypotheses, run one experiment that tells them apart, fix the root cause, prove it with a regression test, and look for siblings.
- Technology
Minimal reproductions, stack traces and structured logs, read-only queries, git bisect, regression tests, and a fresh agent session seeded with a written summary.
- Trade-offs
Methodical debugging is slower to the first change but faster to the right one. Letting the agent loop alone is fast when the bug is shallow and expensive when it isn't; production urgency may call for mitigation first.
The problem
2 min- You · Engineer at StayHubFix number 3 is in: no more errors. Claude says it's done.
- Lina · Senior engineer, your mentorYou drive; I'll only ask questions. Why were they failing?
- You · Engineer at StayHubSomething about exchange rates. Fix 1 set missing rates to 1, fix 2 retried, fix 3 catches the error.
The big idea
4 minEvidence before changes: reproduce, test hypotheses without touching code, and accept only a fix whose cause you can say in one sentence.
Each round, the agent read one error line, guessed, changed code, and the error changed shape. That's a guess loop: it ends when the error is hidden, not when the bug is gone.
How it works
5 minParacetamol brings a fever down while the infection carries on. Fix 3 was a fever pill: the thermometer reads normal because we took it away. Here's the loop that ends at the cause.
- 1Reproduce
A fails every time, with a command to rerun:
pnpm invoices:generate --hotel 4521. Revert the three fixes first. - 2Collect facts
The full stack trace, a failing and a working hotel side by side, and what the 210 have in common.
- 3Hypotheses, no code
A hypothesis is a possible cause plus the evidence that would rule it in or out. Ask for three, and change nothing.
- 4One read-only experiment
A query or one test input that separates the hypotheses. Every code change is an experiment that also changes the system.
- 5Fix the cause, then siblings
The is why it happened, not what you saw. Fix it with a regression test that failed first, then search for the same mistake elsewhere.
- 6Refuse symptom patches
A symptom patch hides or reshapes the error: a catch-all, an invented default, a blind retry. After two failed fixes: summary, revert, fresh session.
Watch a decision being made
4 minYou find the invoice bug; Lina only asks questions
210 GBP/CHF hotels can't generate their September invoice. Find the cause and make sure it can't come back.
- 1Clean slate, then hypotheses
Revert the fixes. Run hotel 4521 (fails) and 1203 (works). "Don't change code. Three hypotheses, and how to tell them apart."
textH1: no GBP/CHF rates at all -> check rate table for GBP rows H2: no rates on some dates -> check failing bookings' dates H3: currency formatting crashes -> render a GBP amount aloneWhy: Hypotheses with distinguishing evidence turn guessing into elimination.
- 2One read-only check
Group the 210 by currency and weekend bookings: all GBP/CHF, all with weekend bookings. The rate table has no Saturday or Sunday rows: rates are published on business days only.
Why: H1 and H3 are ruled out in one step, without changing anything.
- 3Get the rule from the owner
Finance: a Saturday booking uses the last rate published on or before it (Friday's).
Why: A business decision. The agent would pick something plausible; finance knows what accounting expects.
- 4Regression test, fix, siblings
A test for 4521's Saturday booking fails, then passes after the fix. "Find every lookup of a rate by exact date": refunds do it too. Same fix, same kind of test.
Why: A wrong assumption rarely lives in only one place.
In real life
3 minLinux kernel: git bisect
- Widely used to find the first bad commit
- Binary search over history
- Needs a reliable reproduction
Precise when it used to work.
Cloudflare-style: mitigate first
- Public incident write-ups
- First stop the harm (roll back, disable)
- Root cause investigated after
Stop the bleeding, then diagnose.
What goes wrong
3 minThe agent guesses from one line; wrong guesses reshape the symptom.
Reproduction, hypotheses and a read-only check before any change.
The failure continues silently, or becomes a plausible wrong number.
Fail loudly with context; get the rule for missing data from its owner.
Wrong theories in context steer every next attempt.
After two failures: summary of facts, revert, fresh session.
Trade-offs
3 min| Approach | Fits when | You pay |
|---|---|---|
| You drive hypotheses, agent executes | Money, data, guest-facing bugs | Slower to the first change |
| Agent loops alone against a failing test | Shallow, well-reproduced bugs, throwaway branch | Can patch symptoms to green |
| Mitigate first, debug after | Guests or hotels harmed right now | The mitigation's cost |
- Karim · CTOGive Claude production logs and let it debug overnight alone. It's faster than any of us.
- Omar · On-call SREThose logs have guest names and emails. And alone, it patches until the error disappears.
- Karim · CTOSo humans babysit every bug? That doesn't scale.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Whether to mitigate first (hold the batch, roll back, disable a flag) or debug while harm continues.
- The business rule when data is missing (the rate on a Saturday): its owner decides, never the agent.
- When a fix is acceptable: you can state the cause in one sentence and saw a regression test fail first.
Bug: the September invoice fails for ~210 of 10,000 hotels, all billing in GBP or CHF. Reproduce with pnpm invoices:generate --hotel 4521 --month 2026-09; hotel 1203 works. Don't change code yet. Run both and show the outputs. Then give three hypotheses, the evidence for each, and one read-only query to tell them apart. No catch-alls, retries or default rates.
- The bug was reproduced before any change, with a command you can run.
- You can say in one sentence why it happened and why the change stops it.
- Siblings were searched: other code using the same assumption.
- It changes code right after seeing one error line.
- A try/catch that returns an empty or default result.
- A missing rate, price or date set to 0, 1 or "today" to silence the error.
Check yourself
2 minNext episode: Maya's partner-reports page goes live, and on Wednesday a hotel owner writes: "I can see my competitor's revenue."
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Finding the invoice bug with evidence (you drive, Lina asks)
210 GBP/CHF hotels can't generate their September invoice. Find the cause, fix it so finance agrees, and make sure it can't come back, with Claude Code doing the typing.
- 1Clean slate, then reproduce
Revert the three fixes (
git restoreor reset the branch). Ask Claude to run the failing command for hotel 4521 and a working one (hotel 1203) and show both outputs.Why: You can't reason about a bug through three layers of patches. Two runs side by side are your first evidence. Lina: "What's different between those two hotels?"
- 2Ask for hypotheses, not fixes
"Don't change any code. Give me three hypotheses for why 4521 fails and 1203 works, each with the evidence that would rule it in or out, and one read-only check."
textH1: no GBP/CHF rates at all -> check rate table for GBP rows H2: no rates on some dates -> check failing bookings' dates H3: currency formatting crashes -> render a GBP amount aloneWhy: Hypotheses with distinguishing evidence turn guessing into elimination.
- 3Run one read-only check
A query groups the 210 failing hotels by currency and by whether they have weekend bookings. Result: all GBP/CHF, all with weekend bookings. A GBP test hotel with weekday-only bookings works.
Why: H1 and H3 are ruled out in one step. The rate table has no Saturday or Sunday rows: rates are published on business days only.
- 4Get the rule from the owner
Ask finance: which rate applies to a Saturday booking? Answer: the last rate published on or before that date (Friday's).
Why: This is a business decision. An agent would pick something plausible; finance knows what their accounting expects.
- 5Regression test, then fix
"Add a test for hotel 4521's Saturday booking that expects Friday's rate. Show it failing. Then change the lookup to use the last rate on or before the date. Don't touch other tests."
Why: The failing test proves it reproduces the bug; green proves the fix addresses it. You can now say the cause in one sentence.
- 6Hunt siblings
"Find every other place that looks up a rate by exact date." Refunds do: a Sunday refund in CHF would crash too. Same fix, same kind of test.
Why: The wrong assumption ("a rate exists every day") rarely lives in only one place.
Summary for a fresh session (invoice bug):
- Reproduce: pnpm invoices:generate --hotel 4521 --month 2026-09 (fails); hotel 1203 works.
- Known: all 210 failing hotels bill in GBP/CHF and have weekend bookings.
- Ruled out: missing GBP/CHF rates entirely; currency formatting.
- Hypothesis: getRate(date) returns undefined on weekends (no published rate).
- Already tried and reverted: default to 1.0, retry, catch-all.
- Next: confirm with the rate table for 2026-09-05 (a Saturday); finance rule = last rate on or before date.Analogy
- At the doctorPill that just lowers the feverDebugging with an agentA fix that hides the error
- At the doctorQuestions and a temperature checkDebugging with an agentReproduce and collect facts
- At the doctorOne test to tell two causes apartDebugging with an agentA read-only check before any change
- At the doctorTreat the infectionDebugging with an agentFix the root cause
Where it breaks: a doctor sees one patient. A bug in code often hits many places at once, so after the fix you go looking for its twins.
Why it matters
Hiding a bug is worse than leaving it visible.
The evidence disappears. If the code swallows the error, your logs and alerts go quiet. The bug keeps hurting people, and nobody sees it.
New bugs appear. If a missing exchange rate becomes "1", a Swiss hotel's invoice comes out in the wrong amount. An error turned into a wrong number.
The agent gets confused. Every failed attempt stays in its memory for that session. After a few, it starts reasoning from its own wrong ideas.
A careful start feels slower for ten minutes, and gets to the right fix sooner.
Check your understanding
- 1.Collect facts: error, stack trace, logs, what failing cases share
- 2.Run one experiment that separates them
- 3.Reproduce the bug reliably
- 4.Search for siblings of the same mistake
- 5.List hypotheses with distinguishing evidence
- 6.Fix the root cause with a regression test
Practice ladder
Which hypothesis survives?
Debugging Warm-upClaude ran the invoice generator on four test hotels and logged this:
| Hotel | Currency | Booking dates in September | Result |
|---|---|---|---|
| 1203 | EUR | Tue 1, Sat 5, Wed 16 | OK |
| 5120 | GBP | Wed 2, Thu 17 | OK |
| 4521 | CHF | Fri 4, Sat 5 | rate undefined |
| 6002 | GBP | Sun 13 | rate undefined |
Hypotheses: H1 no rates exist for GBP/CHF; H2 no rates exist on weekends; H3 the PDF fails on long hotel names. Trace which survive and why.
Complete the reset summary
Debugging PracticeYou're resetting a looping session on a different bug: some guests see 0 results when searching hotels in Lisbon for dates in December. Claude tried two fixes (raising a timeout; clearing the search cache) that didn't help. Complete the summary for a fresh session:
- Reproduce: ___
- Known: happens for Lisbon, December dates; Porto in December works; Lisbon in November works.
- Ruled out: ___
- Tried and reverted: ___
- Hypotheses: ___ (at least two)
- Next read-only check: ___
Occupancy above 100%
Debugging PracticeThe internal hotel dashboard shows occupancy of 104% for some hotels on some nights. Revenue managers are starting to distrust it. Write the full instruction you'd give Claude Code, and say what you'd need to see before accepting a fix.
Friday 19:00: ship the fix or roll back?
Decision StretchA deploy at 17:00 changed promo-code handling. Since then, guests in Germany get "invalid code" for valid codes; about 40 bookings an hour are affected, some abandon. Claude Code has a candidate fix: it says the cause is "a locale issue in code parsing". You can't fully explain the fix, and it touches the price calculation. Rolling back the 17:00 deploy takes 5 minutes and nothing else shipped since.
Decide what to do now and on Monday. Write: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain to Maya why Claude Code's third fix ("invoices generate now, no more errors") was worse than no fix at all, and how you'd run the debugging instead.