Episode 137 · The deploy that charged nine guests

The agent runtime: production architecture around the model

Thursday, 14:00. A routine deploy restarts StayHub's API, and 212 booking-agent runs die mid-step. By 14:20, nine guests have been charged for rooms that were never booked.

A real AI agent is mostly not the model. It's the system around it that keeps bookings correct when things go wrong.

30 min advancedAgent runtime architecture: gateway, runtime, model, memory, RAG, tools, policies, observabilityDurable agent run state, checkpoints and resumptionTool reliability in agents: timeouts, retries, idempotent side effectsAI gateway responsibilitiesIdempotent write tools under retries and timeoutsStep limits and stopping conditions

Problem
StayHub wants its booking agent to search hotels, book rooms, take payment, update the CRM and email confirmations, for about 5,000 runs a day.
Why the simple system fails
The prototype is a while-loop in an API handler: a model call, a tool call, repeat. A deploy kills runs mid-booking, a payment timeout gets retried into a double charge, one slow tool hangs the request, and nobody can explain what happened.
Engineering concept
Treat the agent as a long-running workflow executed by a runtime: persisted state per step, budgets and deadlines, per-tool timeouts and retry rules, idempotency keys on every side effect, policies before actions, and a trace of everything.
Technology
AI gateway, agent runtime (loop + state store + queue), model, memory store, RAG retriever, tool registry, policy engine, OpenTelemetry-style tracing.
Trade-offs
Durability and safety add latency, storage and engineering effort. A five-second Q&A bot does not need durable execution; an agent that moves money does.

The problem

2 min

Thursday, 14:20. Twenty minutes after a routine deploy.

Omar · On-call SRE
The 14:00 deploy restarted the API. 212 agent runs were in flight. 9 guests were charged, but have no booking.
You · Engineer at StayHub
So the payment tool has a bug?
Lina · Senior engineer, your mentor
The payments were fine. Guess first.

The big idea

4 min

Where a run lives when the server restarts

The big idea

An agent with side effects is a workflow, not a request: persist every step, so a restart resumes instead of forgets.

A chef with orders in their head versus a ticket rail. Crashes, scaling events and deploys happen at any time; the question is where the run is when the process dies.

How it works

5 min

An is the code that runs the loop: it stores state, calls tools with timeouts and retries, checks policies and enforces budgets.

One agent run, with a runtime

1
Queue the run
The API creates a run record and queues it; it doesn't run the loop itself.
2
Workers run the loop
If a worker dies, another resumes the run from the journal.
3
Journal each step
A run journal records each step's input, output and status (). Before a step: already done? Return the saved result.
4
Key every write
An makes doing a write twice the same as once. Use run_id:step_no: the same on every retry and after every restart.
5
Stop at the budget
A : max 20 steps, 90 s, 60k tokens. Past that, the run goes to support with its journal.

Watch a decision being made

4 min

Worked example

Lina turns the loop into a runtime

Make runs survive restarts, make writes safe to retry, and stop runaway loops, using the Postgres StayHub already runs.

1Persist runs and steps

agent_runs and agent_steps tables; workers resume from the last succeeded step.

const done = await steps.find(run.id, stepNo);
if (done?.status === "succeeded") return done.output;
if (stepNo > 20 || Date.now() > run.deadlineAt) throw new BudgetExceeded();

Why: A run you can't see after a crash is a run you can't finish.

2Give every tool a contract
Search 2 s, retried; booking 5 s with a key; payment 10 s, never retried blind; CRM and email queued.
Why: One global timeout is too long for search and too short for payment.
3Treat write timeouts as unknown
If create_payment times out, ask the provider's status with the same key, then continue, retry or compensate.
Why: Retrying blind double-charges; giving up leaves a charge with no booking. Both happened.
4Never let the model pass the amount
The model references the booking; the server computes the price.
Why: An injection or arithmetic slip would become a real charge.

In real life

3 min

Two well-known approaches

Uber: a workflow engine

Built Cadence (open source, later forked as Temporal)
Persists each step's history and resumes after crashes

Fits many long workflows; a platform to operate.

Stripe: every write safe to retry

Its public API accepts an idempotency key: same key, same result
The client still remembers which step it reached

Fits few, well-defined writes.

What goes wrong

3 min

Running the agent loop inside an HTTP request

Runs outlive request timeouts and deploys.

Instead

Persist run and step state; resume on workers.

Letting the model retry writes

It doesn't know if the first call succeeded, and a new call has a new key.

Instead

The runtime owns retries, same key, status check first.

Email and CRM on the critical path

If email is down, the guest sees a failed booking that's actually booked and paid.

Instead

Queue them with their own retries ().

Trade-offs

3 min

Option	Fits when	Costs
In-process loop	Read-only, short runs, prototypes	Runs die with the process
Runtime with persisted state	Side effects, runs over a few seconds	A write per step; your own state machine
Durable workflow engine	Many long multi-step agents	A new platform to learn and run

Monday, 10:00. Post-incident review.

Omar · On-call SRE
Two incidents in a week. Adopt a workflow engine; stop hand-rolling resume logic.
Karim · CTO
A new platform for five people, three weeks before the winter sale? The journal already fixed the deploy problem.
Omar · On-call SRE
And when corporate bookings wait 24 hours for approval?

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Where a run lives: run and step journal in Postgres, executed by queue workers.
The budgets per run (20 steps, 90 s, 60k tokens) and what the guest hears when one is hit.
Each tool's contract, and that write timeouts mean "check status", never a model retry.

Tell Claude Code

Move the booking agent's loop out of `src/agent/handler.ts` into a durable runtime; a deploy killed 212 runs and 9 guests were charged with no booking. Plan first; list tables and states. Persist steps before and after each tool call; workers resume from the first incomplete step; a tool registry with timeout, retry class, key; create_payment keyed run_id:step, 10 s, never retried by the model, status check on timeout; 20 steps, 90 s, 60k tokens. Test: kill a worker mid-payment, one charge.

Check before you accept

The kill-the-worker test: one charge, one booking, a journal showing where it resumed.
The key is derived from run and step, not a fresh UUID per attempt.
A timed-out payment double is followed by a status check, not a second charge.

Red flags

"If the payment fails, try again" left to the model.
Timeout raised to 60 s instead of handling the unknown outcome.
Resuming means re-running the whole run from the start.

Check yourself

2 min

Q1/3Debugging

Logs show a guest was booked twice. The first create_booking call timed out after 5 s, the runtime retried, and both calls succeeded at the provider. What is the most important fix?

How confident are you?

Next episode: the runtime is solid, but a guest just talked the support agent into a 1,840 EUR "loyalty refund". Who decides what the agent is allowed to do?

Finished?

Mark it complete to update your skill map.

Next: Policies and guardrails: what the agent is allowed to do

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

One step, made safe

Worked example

Lina turns the loop into a runtime

The booking agent runs as a while-loop in the API handler. A deploy killed 212 runs and left 9 orphan charges. Make runs survive restarts, make writes safe to retry, and stop runaway loops, using the Postgres StayHub already runs.

1Persist runs and steps

Create agent_runs (status, step count, tokens, deadline) and agent_steps (input, output, status, idempotency key).

sql

CREATE TABLE agent_steps (
  run_id   uuid REFERENCES agent_runs(id),
  step_no  int  NOT NULL,
  kind     text NOT NULL,       -- 'model' | 'tool' | 'approval'
  input    jsonb NOT NULL,
  output   jsonb,
  status   text NOT NULL,       -- 'started' | 'succeeded' | 'failed'
  idempotency_key text UNIQUE,
  PRIMARY KEY (run_id, step_no)
);

Why: A run you can't see after a crash is a run you can't finish. The journal is what makes resuming possible.

2Run the loop on workers
The API creates the run and queues it. Workers execute steps; on startup they pick up runs whose status is running and resume from the last succeeded step.
Why: An HTTP request dies with its process and has a timeout. A queued run outlives both.
3Give every tool a contract
Register each tool with what it changes, its timeout, its retry class and whether it needs approval: search 2 s and retry; booking 5 s with a key; payment 10 s, never retried blind; CRM and email async via a queue.
Why: One global timeout is too long for search and too short for payment. The contract makes the rule explicit and reviewable.
4Key every write
Pass run_id:step_no as the idempotency key to booking, payment, CRM and email.
Why: Retries after timeouts and restarts are certain to happen. The key turns "do it again" into "give me the first result".
5Treat write timeouts as unknown
If create_payment times out, query the provider's status with the same key, then continue, retry or compensate.
Why: Retrying blind can double-charge; giving up can leave a charge with no booking. Both happened to us.
6Add budgets and a deadline
Max 20 steps, a 90 s run deadline (passed down so a late step gets only the time left), and a 60k-token budget. Past any limit, hand the run to support with its journal.
Why: A confused model can search in circles for minutes and dollars. A budget turns that into a clean handover.

Tool	Side effect	Timeout	Retry	Idempotency	Approval
search_hotels	None (read)	2 s	2× with backoff	Not needed	No
create_booking	Holds inventory	5 s	Only with key; on timeout, check status	run_id:step	Yes (guest)
create_payment	Moves money	10 s	Never blind; check provider status	run_id:step	Yes (guest), amount computed server-side
crm_upsert	Updates profile	3 s	5× async via queue	guest_id + booking_id	No
send_email	Sends message	3 s	5× async via queue	message key	No; recipient = verified address

A tool registry entry is a contract: what it changes, how long it may take, whether and how it may be retried.

Interactive simulation

Visual diagram

Guest

AI gateway

Agent runtime

Model

Memory + run state

RAG (policies, FAQs)

Observability

Policy engine

Hotel search

Booking API

Payments

CRM

Click a component

Production architecture of a hotel booking agent. Click components or follow one booking run.

Analogy

A travel agencyAn agent runtime

A travel agencyA brilliant travel agentAn agent runtimeThe model
A travel agencyThe front desk that checks and routes youAn agent runtimeThe AI gateway
A travel agencyA case file a colleague can pick upAn agent runtimeSaved run state
A travel agencyA rulebook: big refunds need a managerAn agent runtimePolicies checked in code
A travel agencyCompany systems with receiptsAn agent runtimeTools with idempotency keys
A travel agencyA logbook the manager can reviewAn agent runtimeObservability and traces

Where the analogy breaks: a human agent remembers yesterday's client. The model remembers nothing between calls, so the runtime has to hand it the whole case file every single time.

Check your understanding

Q1/3Reasoning

Match each responsibility to the component that should own it:

Fail over to another model provider

Stop the run after 20 steps

Refunds above €150 need a manager

Retrieve the cancellation policy text

Remember the guest prefers quiet rooms

How confident are you?

Practice ladder

Step 1 · Trace

Resume a crashed run

Scenario Warm-up

A worker crashed. The journal for run r-77 says:

step	kind	tool	status
1	model	–	succeeded
2	tool	search_hotels	succeeded
3	tool	get_availability	succeeded
4	tool	create_booking	succeeded
5	tool	create_payment	started

A new worker picks up r-77. Trace what it does for steps 1 to 5, and what happens next if the provider says the payment with key r-77:5 succeeded.

Step 2 · Complete

Finish executeStep

Code Practice

Three lines are missing from Lina's executeStep. Fill them in and say what each guarantees.

async function executeStep(run: Run, call: ToolCall) {
  const stepNo = run.stepCount + 1;
  ___                                                        // (1) resume
  if (stepNo > MAX_STEPS || Date.now() > run.deadlineAt) throw new BudgetExceeded(run.id);
  const decision = await policy.check({ run, tool: call.name, args: call.input });
  if (decision.kind === "deny") return { error: "not_allowed" };
  ___                                                        // (2) idempotency key
  await steps.start(run.id, stepNo, call, key);
  const out = await withRetry(() => withTimeout(tool.run(call.input, { idempotencyKey: key }), tool.timeoutMs), tool.retry);
  ___                                                        // (3) journal
  return out;
}

Step 3 · Solve

Email is down, bookings look failed

Debugging Practice

Tuesday, 08:15. The email provider is down for 25 minutes. Maya: "Guests see 'Booking failed, please try again'. Some tried again and now have two bookings."

In the current runtime, send_email is step 7 of the run, synchronous, with a 3 s timeout and no retries; a failed step fails the run.

Explain the cascade and redesign the end of the run.

Step 4 · Decide

Corporate bookings that wait for a manager

Decision Stretch

Next quarter, StayHub adds corporate bookings: the agent prepares a booking, then waits up to 24 hours for the traveller's manager to approve it by email.

Expected: ~1,500 corporate runs a day, ~80% approved within 4 hours.
Today's runtime: Postgres journal, queue workers, 90 s run deadline.
A second agent (group bookings) is planned for the quarter after.

Write a short decision: "We chose X over Y because…, accepting the risk of…". Options: stretch the current runtime (an awaiting_approval status plus a scheduler), or adopt a durable workflow engine.

Explain it yourself

Explain why a production booking agent needs a runtime around the model, and what the runtime does on a single 'book and pay' step.

0 words