Episode 118 · Twenty-seven calls per question

Tool errors, retries and loops that end

Friday, 18:40: the hotel system slows down, and every guest question to the concierge now hits it up to 27 times. Six group bookings already have extra rooms. Lina hands you the incident: "You lead this one."

Tools fail all the time. What you tell the model next decides whether it fixes the problem, asks the guest, or loops forever.

30 min advancedReturning errors to the model as actionable tool resultsWho retries a failed tool: runtime, model or humanIdempotent write tools under retries and timeoutsTransient errors (retry) vs bugs and permanent errors (don't)Timeouts and unknown outcomesPartial failures

Problem
A guest asks to book a room. The PMS times out, the agent retries, the guest gets two bookings, and then the agent loops on an error it can't fix.
Why the simple system fails
Throwing raw exceptions, retrying everything the same way, and leaving retries to the model produce duplicate side effects, wasted tokens and endless loops.
Engineering concept
Classify every failure: who can fix it (the model, the runtime, the user, nobody) decides what the tool result says and who retries. Make writes idempotent and cap the loop.
Technology
Structured error results with is_error and error codes, runtime retries with backoff and jitter, idempotency keys and status lookups, per-item results, step and repetition budgets.
Trade-offs
Hiding errors from the model keeps it focused but can hide information it needs; retrying in the runtime is faster but can amplify outages; strict loop limits stop runaway costs but abandon some solvable tasks.

The problem

2 min

Friday, 18:40. The hotel system (PMS) is having a bad evening.

Omar · On-call SRE
PMS p99 went from 300 ms to 12 s; 8% of calls return 503. Our calls to it are 9× normal.
Omar · On-call SRE
And 6 group bookings have extra rooms. One group that booked 4 now has 7.
Lina · Senior engineer, your mentor
You lead this one. Guess first.

The big idea

4 min

Who can fix this error?

The big idea

For every error, ask who can fix it: that decides what the model is told and who retries.

"Tell the model not to retry" is the wrong instinct: some errors should be retried, just not by the model. And a timeout on a write isn't a failure; it's an unknown you must resolve first.

How it works

5 min

An error result () is a tool result marked is_error, with a short code and one sentence written for the model. Classify every failure:

Classify every failure

1
The model can fix it
Invalid arguments (bad date, unknown room code). Say exactly which field is wrong; it retries with new input.
2
The guest can fix it
Booking not found, room sold out. Tell the model to ask the guest or offer alternatives. Never guess ids.
3
The runtime can fix it
A transient error (503, 429, dropped connection) on a read. The executor retries with : growing, randomised waits. The model hears nothing unless it gives up.
4
Nobody knows yet
A write timed out: an unknown outcome (). Look it up by before doing anything else.
5
Nobody can fix it
Not allowed, over a limit. Say so, don't retry, offer a human.
6
Two rules on top
Batch tools return per-item results, so only failed items get retried. And the runtime ends the loop: a and a handoff when the same call fails twice.

Watch a decision being made

4 min

Worked example

You fix Friday's incident; Lina checks

Calls multiplied 27×, groups with extra rooms, chats looping. Change the executor and loop so it can't happen again.

1Pick one retry owner
HTTP client retries off. The executor retries transient errors on reads only: 3 attempts, backoff with jitter.
Why: Three layers × 3 = 27 calls. One layer = at most 3, and jitter stops every chat retrying at the same instant.
2Resolve writes by key
Every write gets a deterministic key. On timeout, look it up before anything else.
ts
```
if (key && isTimeout(e)) {
  const found = await pms.findByReference(key);
  return found ? ok(found) : retryOnce(tool, args, key); // same key: safe
}
```
Why: Retrying may double-book; reporting failure may make the guest book again. Find out first.
3Report per item
book_rooms returns succeeded (3 booking ids) and failed (the suite, CONFLICT). The retry contains only the suite.
Why: "Failed" when 3 rooms existed is what turned 4-room groups into 7-room groups.
4Cap the loop, hand off well
Max 12 steps; same failing call twice → stop. The handoff lists what already happened: "3 rooms booked: B-9921, B-9922, B-9923".
Why: Termination must not depend on the model, and a guest must always know what they own.

In real life

3 min

Who removes the duplicates?

Stripe: the server dedupes

Publicly documents an Idempotency-Key header on writes
Retrying with the same key returns the first result
Callers can retry a timeout safely

Easiest for callers, if the server supports it.

Amazon SQS standard queues: the consumer dedupes

Publicly documented as at-least-once: a message can arrive twice
Consumers handle duplicates themselves

Works with anything, but every consumer must remember what it did.

What goes wrong

3 min

Returning an empty result when a tool fails

The model can't tell "no rooms" from "search is down", and states a false fact.

Instead

is_error with a code; empty only when the answer really is empty.

Stack traces in the tool result

Wastes tokens, can leak internals, gives the model nothing to act on.

Instead

A code and one sentence for the model; details in logs.

Retrying a write after a timeout

The first attempt may have committed: two bookings or charges.

Instead

Look the outcome up by key or reference first.

Trade-offs

3 min

Failure	Who fixes it	Runtime retries?
Invalid arguments	Model	No
Transient error on a read	Runtime	Yes, 3× with backoff + jitter
Timeout on a write	Runtime	Only after a check by key
Partial batch failure	Model + guest	Only the failed items

Monday, 10:00. The incident review.

Maya · Product manager
The 12-step budget cuts off big group bookings. 1 chat in 200 now ends with "a colleague will call you".
Omar · On-call SRE
Before, 1 chat in 200 looped to 40 steps on a broken PMS. Keep it tight.
Maya · Product manager
Those were broken loops. These are real guests with real money.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which layer owns retries: only the tool executor, not the HTTP client, the executor and the model (3 × 3 × 3 = 27).
Which tools may be retried automatically: reads yes; writes only with an idempotency key and a status check first.
What the model is told per error kind (fix arguments, ask the guest, or don't retry and offer a human) and when the loop stops (12 steps).

Tell Claude Code

Friday's PMS slowdown showed three retry layers stacking to 27 calls per question, and `book_rooms` returning "error" when 3 of 4 rooms succeeded. Make the tool executor the only retry owner (3 attempts, backoff with jitter, reads only). Writes send an idempotency key and, after a timeout, check status by key first. `book_rooms` returns per-room results. Errors get a code, one sentence and a `retryable` flag. Step budget 12; hand off when a failing call repeats. Plan first, then show tests for a write timeout and a 3-of-4 batch.

Check before you accept

Search the diff for every retry setting: exactly one layer retries.
The timeout test on create_booking returns the existing booking, not a second one.
A failed search gives the model an error, not an empty list it reads as "no rooms".

Red flags

Retries added in yet another layer "to be safe".
A handler that catches exceptions and returns [] as if the search worked.
The step limit raised so looping chats can finish.

Check yourself

2 min

Q1/3Scenario

create_booking times out after 10 seconds. What should the executor do first?

How confident are you?

Next episode: three StayHub teams have each wired the same hotel system into their own AI tools, and a single API change just broke two of them.

Finished?

Mark it complete to update your skill map.

Next: MCP: one protocol between AI apps and the world

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Fixing Friday, one layer at a time

type ToolError = {
  code: "INVALID_ARGUMENT" | "NOT_FOUND" | "PERMISSION_DENIED" | "CONFLICT"
      | "UNAVAILABLE" | "RATE_LIMITED" | "PARTIAL_FAILURE";
  message: string;     // one or two sentences, written for the model
  retryable: boolean;  // may the MODEL try again with different input?
  fix?: string;        // what to do next
};
// What the model receives:
// { type: "tool_result", tool_use_id, is_error: true,
//   content: '{"code":"INVALID_ARGUMENT","message":"check_out is before check_in.",
//              "retryable":true,"fix":"Ask the guest to confirm the dates."}' }

is_error tells the model this is a failure, not data. retryable and fix steer the next step.

Worked example

You fix Friday's incident (Lina checks)

The PMS is slow, calls are multiplied 27×, groups get extra rooms, and some chats loop. Change the tool executor and loop so this can't happen again.

1Classify each failure
Map every error the PMS can return to a code: INVALID_ARGUMENT, NOT_FOUND, CONFLICT, PERMISSION_DENIED (permanent) or UNAVAILABLE, RATE_LIMITED (transient). Timeouts on writes become "unknown".
Why: Who can fix it decides what to do. You can't choose a retry policy for an error you haven't named.
2Pick one retry owner
Turn off retries in the HTTP client. The executor retries transient errors on reads only: 3 attempts, exponential backoff with jitter. The model is told only when they're used up.
Why: Three layers × 3 retries = 27 calls. One layer = at most 3, and jitter stops all chats retrying at the same instant.
3Resolve writes by key
Every write gets a deterministic key (conversation + tool + arguments). On timeout, look the key up with the PMS (or our reference field for old PMSs) before doing anything.
ts
```
if (key && isTimeout(e)) {
  const found = await pms.findByReference(key);
  return found ? ok(found) : retryOnce(tool, args, key); // same key: safe
}
```
Why: A timeout means "we don't know". Guessing either way is wrong: retrying may double-book, reporting failure may make the guest book again.
4Report per item
book_rooms returns succeeded and failed lists with booking ids and codes, plus a fix line. The retry contains only the failed items.
Why: Saying "failed" when 3 rooms exist is what turned 4-room groups into 7-room groups.
5Cap the loop in code
Max 12 model calls per turn. Same tool + same arguments failing twice → stop. Three UNAVAILABLE from the PMS in one chat → stop.
Why: Models often keep trying variations of a call that can't succeed. Termination must not depend on the model.
6Hand off gracefully
The handoff summary lists what already happened ("3 rooms booked: B-9921, B-9922, B-9923") and offers a callback.
Why: The worst outcome is a guest who doesn't know what they own. Writes that succeeded must always be reported.

Failure	Example	Who fixes it	Runtime retries?	What the model is told
Invalid arguments	Bad date, unknown enum value	Model	No	Which field is wrong and what's allowed
Not found	No reservation LIS-48213 for this guest	Guest	No	Ask the guest to check the number; don't guess ids
Permission / policy	Refund over the limit	Nobody (or a human)	No	Not allowed; don't retry; offer a handoff
Conflict	Room just sold out	Model + guest	No	Search again and offer alternatives
Transient (read)	503, 429, connection reset	Runtime	Yes, 3× with backoff + jitter	Nothing, unless retries run out
Timeout on a write	create_booking > 10 s	Runtime	Only after a check by key	The real outcome, never a guess
Partial failure	3 of 4 rooms booked	Model + guest	Only failed items	Per-item results

Permanent errors are never retried unchanged. Transient errors are retried by code, not by spending model calls.

Interactive simulation

Visual diagram

LLM

Agent loop

Tool executor

Idempotency store

PMS API

Payments

Logs & traces

Click a component

Where each kind of failure is handled. Most never reach the model; some must.

Analogy

A temp and the back officeAn agent and its tools

A temp and the back office"Wrong form: 31 February"An agent and its toolsAn error the model can fix
A temp and the back officeClerk quietly reprints a jammed pageAn agent and its toolsYour code retries a brief failure
A temp and the back office"Ask the guest for their number"An agent and its toolsAn error the user must fix
A temp and the back officeSerial number on every booking formAn agent and its toolsAn idempotency key stops duplicates
A temp and the back officeAfter three rejections, call the managerAn agent and its toolsA step limit and a human handoff

Where the analogy breaks: a temp gets frustrated and stops. A model never gets tired, so it will happily retry forever unless your code says stop.

Why it matters

In real life, tool failures are not rare. Payment providers, booking systems and email services all have bad days. And models get small details wrong now and then. Over a long conversation, it adds up.

Get it wrong and you pay three ways.

Money and trust: two bookings, two charges, two emails. Guests remember those.

Time and tokens: every retry is another full model call. An agent looping ten times costs about ten times more, and still fails.

Wrong answers: if a broken search returns an empty list, the model says "no rooms available". The truth was "search is down".

Check your understanding

Q1/3Reasoning

Match each failure to who should handle it:

check_out before check_in

PMS returns 503 on a search

No reservation under that number for this guest

Refund above the agent's limit

How confident are you?

Practice ladder

Step 1 · Trace

Trace a write timeout

Debugging Warm-up

After your fix, the executor log for one chat shows:

19:02:11.004  create_booking  quote=Q-912  key=c-77:create_booking:9f3a  attempt 1
19:02:21.010  create_booking  TIMEOUT after 10 s
19:02:21.050  pms.findByReference(c-77:create_booking:9f3a)  -> { booking_id: "B-1204", status: "confirmed" }

Trace what the executor does next, what the model receives, and what the guest is told. Then say what would have happened with Friday's old code.

Step 2 · Complete

Fill in the error results

Design Practice

Lina's table for change_reservation_dates has three rows missing. For each, give the code, whether the runtime retries, and the one-line message the model receives.

Situation	Code	Runtime retries?	Message for the model
check_out is before check_in	INVALID_ARGUMENT	No	"check_out (2026-06-12) is before check_in (2026-06-14). Ask the guest to confirm the dates."
Reservation belongs to another guest	___	___	___
PMS returns 503 three times in a row	___	___	___
The new dates are fully booked	___	___	___

Step 3 · Solve

30 reminder emails, 4 failures

Design Practice

The staff assistant has a tool send_checkin_reminders({ date }) that emails every guest arriving on that date. For tomorrow it's 30 guests. The email provider accepted 26 and rejected 4: two invalid addresses, one rate-limit (429), one timeout.

Design the tool's result, decide who retries what, and say how a second call to the tool must behave so no guest gets two emails.

Step 4 · Decide

Tell the guest it's slow?

Decision Stretch

During PMS slowdowns, the executor's read retries add up to 6 s before the model answers. Two options:

A: keep hiding transient errors from the model; the guest just waits.
B: after the first failed attempt, send the model a short note ("availability is slow today, retrying") so it can tell the guest while the executor keeps retrying.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain how your agent should handle a create_booking call that times out, and why.

0 words