Skip to content
Episode 114 · €4,800 for a question

Human in the loop: approvals, thresholds and escalation

Thursday, 15:05: a guest asked the guest-service agent "what's your refund policy?" about her group booking, and it refunded her €4,800. "Always ask before refunding" was right there in its prompt.

Agents that act in the real world need people at the right moments, not at every step. Here's how to choose those moments.

30 min advancedHuman confirmationApproval gates for side effectsConfidence thresholds and escalation to humansDesigning confirmation UX that people actually readLeast privilege for agentsIdempotency keys & dedupe tables
  1. Problem

    A guest-service agent can rebook rooms, issue refunds and email guests. One afternoon it refunds €4,800 to a guest who asked “what's your refund policy?”.

  2. Why the simple system fails

    Either the agent acts alone on everything (fast, until it's catastrophically wrong), or every step needs a click (safe on paper, but humans rubber-stamp hundreds of prompts a day and the agent is no faster than doing it by hand).

  3. Engineering concept

    Put humans where risk and uncertainty are highest: classify actions by reversibility and blast radius, gate the risky ones in code, bind each approval to the exact effect, auto-handle confident low-risk cases, and escalate the rest with full context.

  4. Technology

    Tool risk tiers and policy rules, approval records tied to argument hashes, dry runs, durable paused runs resumed by webhooks, calibrated confidence thresholds, handoff queues with SLAs, and approval metrics.

  5. Trade-offs

    Every gate adds latency and human cost; every auto-approved action adds risk. Thresholds trade automation rate against error rate, and too many prompts create approval fatigue that makes the gates meaningless.

01

The problem

2 min
Thursday, 15:05. The guest-service agent handles ~2,400 messy post-booking cases a day.
  1. A guest · StayHub customer
    What's your refund policy for my group booking?
  2. Omar · On-call SRE
    The agent called issue_refund for €4,800: 12 rooms, non-refundable rate. The prompt says "always ask before refunding".
  3. Maya · Product manager
    How do we make sure it never happens again?
  4. Lina · Senior engineer, your mentor
    You lead the diagnosis. Guess first.
02

The big idea

4 min
A rule in a prompt is not a gate
The big idea

A gate is code, not a prompt line. Put people where risk and uncertainty are highest, and have them approve the exact effect.

The question isn't "should there be a human?". It's which actions, at what threshold, approved by whom, shown how. With the right card, nobody approves €4,800 for a question.

03

How it works

5 min
Where people belong in the guest-service agent
  1. 1
    Tier every action

    By reversibility and blast radius (how much damage a wrong action does: one guest, one hotel, everyone). Lookups run; rebook needs the guest's confirmation; refunds over €200 need staff; bulk emails need two people.

  2. 2
    Gate in code

    An approval gate () lives in the tool executor: no matching, unexpired approval record, no refund. Whatever the prompt says.

  3. 3
    Approve the exact effect

    The card shows amount, card, booking, the policy it breaks, and "irreversible". The approval is bound to a hash of those exact arguments and expires after 10 minutes.

  4. 4
    Calibrate the threshold

    A confidence threshold () set from 2,000 labelled cases: above it, auto-handle low-risk actions (≥ 98% correct); below it, ask the guest. Never the model's own "I'm sure".

  5. 5
    Pause durably, escalate well

    Paused runs are saved and resumed by an approval webhook. Escalation () hands a person the conversation, what was tried and a recommendation.

04

Watch a decision being made

4 min
Worked example

Lina checks your redesign of the agent's controls

Make a €4,800 mistake impossible without drowning staff in approvals, for ~2,400 cases a day.

  1. 1Tier every tool

    lookup_booking automatic; rebook guest confirms; issue_refund ≤ €200 in policy guest confirms, else staff; bulk send_email two-person approval.

    Why: Approval effort should follow risk: most actions are safe, a few are irreversible or reach many people.

  2. 2Gate in the executor

    The tool runner refuses without an approval matching the arguments' hash, unexpired, from a role allowed that amount.

    ts
    const hash = sha256(canonical({ amount, bookingId, cardId }));
const a = await approvals.find({ tool: "issue_refund", hash });
if (!a || a.expiresAt < now() || a.maxAmount < amount)
  return pauseForApproval({ amount, bookingId, cardId }, hash);

    Why: A gate in code can't be skipped, misread or talked out of.

  3. 3Fix the misreading too

    A calibrated intent score: below the threshold, the agent asks "Would you like to cancel, or just know the policy?"

    Why: The same misreading now ends as a harmless clarifying question.

  4. 4Measure attention, not clicks

    Staff approvals ~150 a day (~6%), ~40 s each; wrong refunds ≈ 0. Lina: "Checked."

    Why: Forty seconds means people are reading; the time goes where judgement changes the outcome.

The whole thing in one line: Tiers decide where people look, code enforces the gate, and binding approvals to exact arguments makes what was approved exactly what runs.
05

In real life

3 min
Where should the person be?

Tesla FSD (Supervised): watch all the time

  • The driver must supervise continuously, ready to take over
  • A human in the loop for every moment
  • Relies on sustained attention, which people are bad at

Simple to state, vulnerable to fatigue.

Waymo: autonomy, people when asked

  • Drives without a human driver where it operates
  • Has publicly described remote staff answering the car's questions in unusual situations
  • People are pulled in where the system is unsure

Autonomy where confident, escalation where not.

06

What goes wrong

3 min
Gating in the prompt only

The model can skip, misread or be talked out of the rule.

Instead

The tool executor refuses without a valid approval record.

Approving intent instead of effect

"Refund?" hides amount and recipient; people approve something else than what runs.

Instead

Show the exact effect; bind the approval to the arguments' hash.

Trusting the model's stated confidence

"I'm 95% sure" can be right far less often.

Instead

Calibrate thresholds on labelled cases.

07

Trade-offs

3 min
Staff lineApprovals/dayYou pay
Every refund over €50≈180Slow refunds; attention slips again
Over €200 or out of policy≈50Small wrong refunds slip below the line
Only over €500FewA misread can send up to €500 with no human
Monday, 11:00. Of ~600 goodwill refunds a day, 70% are ≤ €50 and 8% are over €200.
  1. Omar · On-call SRE
    Staff approve every refund over €50. After €4,800, I want people on every real amount.
  2. Maya · Product manager
    Guests waiting hours for €60? Make it €500.
  3. Omar · On-call SRE
    At €500 a misread €450 goes straight out.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • The risk tier of every tool: what runs freely, what the guest confirms, what needs staff, what the agent never gets.
  • The money thresholds (where the staff line sits is a business decision) and the auto-handling threshold, set from 2,000 labelled cases.
  • What an approval card shows (the exact effect from a dry run) and how approvals are bound: argument hash, 10-minute expiry.
Tell Claude Code
The guest-service agent refunded €4,800 to a guest who only asked about the policy; the only gate was a prompt line. Add a policy engine in the runtime that every write tool must pass: `lookup_booking` allowed, `rebook` and `issue_refund` ≤ €200 in policy ask the guest to confirm the dry-run result, anything larger or out of policy pauses the run for staff. Approvals store a hash of the exact arguments and expire after 10 minutes; execution re-checks them and uses an idempotency key. Rules in versioned config. Plan first.
Check before you accept
  • Replay "what's your refund policy?": no refund runs, or it's blocked and escalated.
  • Approve €86, then make the model call issue_refund with €860: execution is refused.
  • A staff card shows amount, recipient, booking and the policy deviation, not just "Refund?".
Red flags
  • Approval enforced by a system prompt line.
  • Every action sent to staff "to be safe".
  • An approval stored as approved: true, so the arguments can change after the human said yes.
09

Check yourself

2 min
Q1/3Debugging
A guest approves “Refund €86”. In the next step, after reading an email the guest forwarded, the model calls issue_refund({amount: 860}) with the same approval id. What should happen?
How confident are you?

Next episode: the last one in this part of the course. You design StayHub's booking agent end to end, for six languages and WhatsApp, and Lina only checks your answers.

Finished?

Mark it complete to update your skill map.

Next: Design a hotel booking agent end to end
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Gates, hashes and thresholds
Worked example

You redesign the guest-service agent's controls

Make a €4,800 mistake impossible without drowning staff in approvals, for ~2,400 cases a day. Lina reviews your design at the end.

  1. 1Tier every tool

    lookup_booking: read, automatic. rebook: reversible, guest confirms the exact change. issue_refund: ≤ €200 within policy → guest confirms; > €200 or out of policy → staff. send_email to many guests: two-person approval.

    Why: Approval effort should follow risk. Most actions are safe; a few are irreversible or reach many people.

  2. 2Gate in the executor

    The tool runner refuses any gated tool without an approval record matching the hash of its arguments, not expired (10 minutes), from a role allowed to approve that amount.

    ts
    const hash = sha256(canonical({ amount, bookingId, cardId }));
const a = await approvals.find({ tool: "issue_refund", hash });
if (!a || a.expiresAt < now() || a.maxAmount < amount)
  return pauseForApproval({ amount, bookingId, cardId }, hash);

    Why: A gate in code can't be skipped, misread or talked out of, and binds what was approved to what runs.

  3. 3Show the exact effect

    The approval card shows amount, card, booking, the policy it touches, the agent's reason and "irreversible".

    Why: The old card said "Refund?": people approved something different from what executed.

  4. 4Calibrate the threshold

    Label 2,000 real cases. Pick the intent-classifier score above which auto-handled cases are ≥ 98% correct. Below it, the agent asks the guest a clarifying question ("Would you like to cancel, or just know the policy?").

    Why: The model's own "I'm sure" isn't calibrated. A threshold from labelled data is.

  5. 5Pause durably, escalate well

    Paused runs are saved; approvals arrive by webhook and resume them. Escalations carry the conversation, what was tried and a recommendation. Lina: "Checked." Staff approvals ~150 a day (~6%), ~40 s each; wrong refunds ≈ 0.

    Why: Approvals take minutes; a request held open would time out. A good escalation saves the person from re-reading everything.

The whole thing in one line: You tiered tools by risk, gated refunds in code with approvals bound to the exact arguments, showed the full effect, calibrated when to act alone, and made pauses durable: from 2,400 reflex clicks to ~150 approvals that each get real attention.
ActionReversible?Blast radiusControl
lookup_bookingYes (no effect)NoneAutomatic
rebook (same price)MostlyOne guestGuest confirms the exact change
issue_refund ≤ €200, in policyNoOne guest, smallGuest confirms; logged
issue_refund > €200 or out of policyNoOne guest, largeStaff approval bound to arguments
send_email to many guestsNoMany guestsTwo-person staff approval
Sort by reversibility and blast radius; put people where both are high.
11

Interactive simulation

12

Visual diagram

Click a component

Where humans sit in a guest-service agent. Click components for details.

13

Analogy

  • Your banking appChecking your balanceHuman in the loopRead actions, no approval
  • Your banking appPaying a known friend 20 eurosHuman in the loopLow-risk action, auto-approved
  • Your banking appConfirming 3,000 euros to a new payeeHuman in the loopApproval gate for that exact action
  • Your banking appA 3 a.m. transfer held, fraud team callsHuman in the loopLow confidence leads to escalation
  • Your banking appConfirming every coffee without lookingHuman in the loopApproval fatigue

Where the analogy breaks: your bank's rules are fixed and tested for years. An agent's mistakes are less predictable, so you watch approval rates and adjust the gates over time.

14

Why it matters

Agents make mistakes rarely, but confidently. An approval turns that mistake into a cheap "no", instead of a refund dispute, a double booking or an awkward email to 3,000 guests.

But people are slow, costly and get tired. A staff approval can take hours, so the agent must pause and resume later. If 40% of cases need a human, you've built an expensive suggestion box.

And fatigue is real. Approve 200 requests a day where 199 are fine, and you stop reading. The one that matters gets the same quick click.

Good design puts people where they matter most, so you get more automation and more safety.

15

Check your understanding

Q1/3Decision
Match each agent action to an appropriate gate:
Search availability for the guest's dates
Hold a room for 15 minutes
Move the guest's booking to new dates
Refund €1,500 outside policy
Email all guests of a hotel
How confident are you?
16

Practice ladder

Step 1 · Trace

Sort six actions into tiers

Decision Warm-up

For each action the guest-service agent might take, give its tier (automatic, guest confirms, staff approves, two-person staff approval) and one line of why.

  1. Look up a guest's booking and its cancellation terms.
  2. Hold a room for 15 minutes while the guest decides.
  3. Move a booking to other dates at the same price.
  4. Refund €35 for a broken kettle, within the goodwill policy.
  5. Refund €640 on a non-refundable rate.
  6. Email all 180 guests of Hotel Ribeira about a pool closure.
Step 2 · Complete

Finish the approval gate

Code Practice

Complete the gate in the tool executor. It must: (1) bind the approval to the exact arguments, (2) reject expired approvals, (3) check the approver may approve this amount, and (4) make the refund safe to retry.

async function executeRefund(args: { amount: number; bookingId: string; cardId: string }) {
  const hash = ___;                                   // (1)
  const approval = await approvals.find({ tool: "issue_refund", hash });
  if (!approval) return pauseForApproval(args, hash);
  if (___) return pauseForApproval(args, hash);       // (2) expired
  if (___) return escalate(args, "approver limit");   // (3) role limit
  return payments.refund(args, { idempotencyKey: ___ }); // (4)
}
Step 3 · Solve

Escalation for an out-of-policy rebooking

Design Practice

A guest's flight was cancelled; she asks to move her non-refundable Porto booking (3 nights, €420) to next month. Policy says non-refundable bookings can't be moved, but hotels often agree in cases like this. The hotel's front desk answers within ~20 minutes during the day; it's 23:40 now.

Design what the agent does: what it tells the guest, what the escalation contains, how the run waits, and what happens if nobody answers by morning.

Step 4 · Decide

Set the auto-handle threshold

Decision Stretch

The intent classifier scores how sure it is that a message is a refund request. On 2,000 labelled cases:

scorecasescorrect
≥ 0.95900891 (99%)
0.85–0.95600564 (94%)
0.70–0.85300246 (82%)
< 0.70200120 (60%)

Auto-handling means the agent proceeds to the refund flow (refunds ≤ €200 still need the guest's confirmation of the exact amount). Below the threshold, the agent asks a clarifying question first. The goal: auto-handled cases ≥ 98% correct.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

17

Explain it yourself

Your CEO asks: “Can't we just make a human approve everything the AI does?” Explain why that's not the safest design, and what you'd do instead.

0 words