Episode 42 · Charged, but no booking

Partial failures: when part of the system breaks

Friday, 19:05: the payment provider slows down for 12 minutes and 140 checkouts time out. By 19:30, 37 guests have been charged twice and 9 have been charged with no booking at all.

Once a request crosses a network, "it worked" and "it failed" get a third sibling: "no idea". Here's how to live with it.

30 min intermediatePartial failuresFallacies of distributed computingTimeouts and unknown outcomesIdempotencyTransactional outboxThe cost of distribution

Problem
A booking needs your service, your database and a payment provider. Any one of them can fail while the others keep going.
Why the simple system fails
Code written as if it were one program assumes a call either works or throws. Over a network, the call can succeed and the reply can vanish, so naive code double-charges or loses bookings.
Engineering concept
Treat every remote call as having three outcomes: success, failure, unknown. Resolve unknowns from the source of truth, make operations idempotent, and own every inconsistency.
Technology
Timeouts, idempotency keys, status lookups, transactional outbox, reconciliation jobs.
Trade-offs
Short timeouts (fast failure, more unknowns) vs long timeouts (fewer unknowns, blocked resources); automatic recovery vs human review.

The problem

2 min

Friday, 19:30. Twenty-five minutes after the payment provider slowed down.

Maya · Product manager
37 guests charged twice, 9 charged with no booking. Support is drowning.
Omar · On-call SRE
The provider was slow for 12 minutes. Our timeout is 8 s. 140 calls timed out.
You · Engineer at StayHub
But every one of those guests saw "Payment failed". How did they get charged?

The big idea

4 min

One timeout, three stories

The big idea

Over a network a call has three outcomes: succeeded, failed, or unknown. Never guess the third: write down the intent, ask again by key, and reconcile.

You post a deposit cheque and hear nothing for two weeks. Post a second? If the first arrived, you've paid twice. You check your bank statement, quoting the cheque number. A is the silence; the provider is your bank.

How it works

5 min

A partial failure is when some parts did their job and others didn't, and no single part knows the whole truth. A timeout is how long you wait before giving up; giving up doesn't undo what the other side did. Raising it only makes unknowns rarer, and holds our threads longer.

A checkout that never guesses

1
Write down the intent
Save the attempt as PENDING_PAYMENT with key bk_77, before any money moves.
2
Charge with the key
Call the provider with an (bk_77): a repeat with the same key can't charge twice.
3
Sort into three
Succeeded → confirm. Declined → release the room. Timeout or network error → unknown outcome.
4
Tell the truth
For unknown, the guest sees "Confirming your booking…", and the room stays held.
5
Find out
A recovery job asks the provider about bk_77 a minute later and confirms or releases. Nightly reconciliation compares our bookings with the provider's charges.

Friday again: the 140 timeouts become 140 "Confirming…" screens. A minute later the job finds 46 were charged and confirms them; 94 are released. No double charges, no charges without a booking. The cost: some guests wait a minute.

Watch a decision being made

4 min

Worked example

Lina rebuilds Friday's checkout

Redesign checkout so a timeout, a crash or a slow provider can never leave a guest charged twice or charged for nothing.

1Name the gaps, classify each failure
Hold → charge → save → publish → email: mark every gap where a crash or timeout stops us. Refused before sending: not charged. Declined: not charged, final. Timeout after sending: unknown. Success, then our crash: charged, state missing.
Why: Only one of these is a real failure. The old code put them all in "failed", and that let guests pay twice.
2Write the intent first, reuse one key
Save PENDING_PAYMENT with key bk_77 before calling the provider; every attempt for this booking uses the same key.
ts
```
} catch (err) {
  result = isTimeoutOrNetworkError(err) ? "unknown" : "failed";
}
if (result === "unknown") return { state: "PENDING_PAYMENT" }; // "confirming…"
```
Why: A crash leaves a row a job can find; the key lets the provider recognise a repeat.
3Show the truth to the guest
On unknown: "Confirming your booking…", room held. Never "failed" unless we know it failed.
Why: "Failed" invites a second payment. "Confirming" is true, and buys the recovery job a minute.
4Give every leftover an owner
Recovery job every minute: look up stuck attempts by key. Nightly reconciliation: compare every provider charge with every booking; alert on gaps.
Why: The job is fast but only sees attempts we recorded; reconciliation starts from the provider's list and catches bugs and lost webhooks.

In real life

3 min

Asking versus being told

Stripe: retry with the same key

Publicly documents idempotency keys: the same key returns the original result
The caller resolves an unknown itself, right away

Fits when you want the answer during or just after the request.

Adyen: wait for the notification

Publicly documents webhooks as the way to learn a payment's final result
Notifications can be delayed or missed, so you still need a check

Fits flows slow by nature (bank redirects, 3-D Secure).

What goes wrong

3 min

Retrying non-idempotent calls automatically

Clients and SDKs often retry by default; a retried charge without a key is a second charge.

Instead

A key on every state-changing call; check what libraries retry.

No timeout at all

A hung dependency holds threads forever; you fail too.

Instead

Explicit timeouts, shorter than your caller's.

Trusting error rates to catch inconsistencies

A crash after a successful charge may log nothing.

Instead

Reconcile regularly; alert on stuck states.

Trade-offs

3 min

Approach	Fits when	You pay
Short timeouts (1–3 s)	Usually-fast dependency, solid recovery	More unknowns to resolve
Long timeouts (10–30 s)	Slow by nature, off the request path	Threads pile up in incidents
Accept, then resolve	Correctness beats an instant answer: payments	More states, recovery jobs

Monday, 10:00. The new checkout design review.

Maya · Product manager
"Confirming…" for up to a minute? Guests hate that. Wait 30 seconds and give them a real answer.
Omar · On-call SRE
At peak that's 2 a second × 30 s = 60 threads stuck, plus re-clicks, out of 200. Search and login share them.
Maya · Product manager
And with 8 seconds, some guests stare at a pending screen.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

How each failure is classified: refused, declined, unknown, or charged-then-crashed. An agent tends to lump them into "failed".
What the guest sees for unknown ("Confirming…", room held), and one idempotency key per attempt, reused on every retry.
Who owns leftovers: the every-minute recovery job and the nightly reconciliation.

Tell Claude Code

A provider slowdown charged 37 guests twice and 9 with no booking. Plan first, no edits until I approve. Before calling the provider, save the attempt as PENDING_PAYMENT with an idempotency key every retry reuses. Three outcomes: succeeded (confirm), failed (release), unknown on timeout or network error (keep the hold, return "confirming"). A job every minute looks up stuck attempts by key; a nightly job compares charges with bookings and alerts. Test a timeout after the charge, and a crash before the confirm.

Check before you accept

In the catch, a timeout leads to "unknown" and keeps the hold.
PENDING_PAYMENT is written before the call, and the key is passed every time.
The two failure tests end with one charge and one confirmed booking; reconciliation alerts rather than silently fixing.

Red flags

Any error, including a timeout or a 500, shown as "payment failed, try again".
A retry with a new key, or none.
"Done" because the error rate is green.

Check yourself

2 min

Q1/3Reasoning

Your call to the payment provider times out after 8 seconds. What do you actually know?

How confident are you?

Next episode: a guest books a room, refreshes the page, and her booking has vanished. It's in the database. So why can't she see it?

Finished?

Mark it complete to update your skill map.

Next: CAP, PACELC and consistency

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Never guess: record, ask, reconcile

The pattern is record intent, act with a key, resolve unknowns. Here's the whole checkout, then the recovery job that resolves what the checkout couldn't:

export async function confirmBooking(attempt: BookingAttempt) {
  await db.bookings.upsert({ key: attempt.key, state: "PENDING_PAYMENT", amount: attempt.amount });

  let result: "succeeded" | "failed" | "unknown";
  try {
    const charge = await psp.charges.create(
      { amount: attempt.amount, currency: "EUR", source: attempt.card },
      { idempotencyKey: attempt.key, timeoutMs: 8_000 },
    );
    result = charge.status === "succeeded" ? "succeeded" : "failed";
  } catch (err) {
    result = isTimeoutOrNetworkError(err) ? "unknown" : "failed";
  }

  if (result === "succeeded") return db.bookings.confirm(attempt.key);   // + outbox event, same transaction
  if (result === "failed") return db.bookings.markFailed(attempt.key);    // + release room hold
  return { state: "PENDING_PAYMENT" }; // the guest sees "confirming…"; the recovery job resolves it
}

// Runs every minute
export async function resolveStuckAttempts() {
  for (const a of await db.bookings.findStuck({ state: "PENDING_PAYMENT", olderThan: "1 minute" })) {
    const charge = await psp.charges.findByIdempotencyKey(a.key);
    if (charge?.status === "succeeded") await db.bookings.confirm(a.key);
    else if (!charge || charge.status === "failed") await db.bookings.markFailed(a.key);
    // still processing: leave it for the next run
  }
}

Three outcomes, three branches. The timeout branch does not guess.

Worked example

Lina rebuilds Friday's checkout

A slow payment provider caused 37 double charges and 9 charges without a booking. Redesign checkout so a timeout, a crash or a slow provider can never leave the guest charged twice or charged for nothing.

1List the steps and the gaps
Checkout does: hold room → charge → save booking → publish event → email. Mark every gap between two steps where a crash or a timeout could stop us.
Why: Each gap is a place where one system has changed and another hasn't. You can't protect a gap you haven't named.
2Classify each failure
Connection refused before sending: definitely not charged. Card declined (4xx): not charged, final. Timeout after sending: unknown. Success reply, then our crash: charged, our state missing.
Why: Only one of these is a real failure. The old code put all of them in the "failed" bucket, and that's what let guests pay twice.
3Write the intent first
Save PENDING_PAYMENT with key bk_77 before calling the provider. Use the same key for every attempt at this booking.
Why: A crash now leaves a row a job can find. The key lets the provider recognise a repeat instead of charging again.
4Show the truth to the guest
On unknown, show "Confirming your booking…" and keep the room held. Never show "failed" unless we know it failed.
Why: "Failed" invites a second payment. "Confirming" is true, and it buys the recovery job a minute.
5Give every leftover an owner
Recovery job every minute: look up stuck attempts by key, confirm or release. Nightly reconciliation: compare every provider charge with every booking, refund or repair the gaps, alert if any.
Why: The recovery job handles the common case fast. Reconciliation catches what it misses: a bug, a lost webhook, a key that never got saved.

Situation	Outcome	Safe reaction
Connection refused before sending	Definitely not processed	Retry (with backoff)
HTTP 4xx (e.g. 402 card declined)	Processed, rejected	Don't retry; tell the guest
HTTP 5xx	Usually not processed, but not guaranteed	Retry only with the same idempotency key
Timeout after sending	Unknown	Look up by idempotency key, then decide
Success reply, then our crash	Remote succeeded, our state missing	Recovery job finishes from the recorded intent

Classify the failure before reacting. Only one row is truly "failed".

Interactive simulation

Visual diagram

Guest app

Booking service

Bookings DB

Payment provider

Recovery job

Nightly reconciliation

Click a component

The payment succeeded, the booking service crashed before saving. Recovery asks the payment provider by idempotency key instead of guessing.

Analogy

Paying a deposit by postCalling a service

Paying a deposit by postPosting a cheque to the hotelCalling a serviceSending a payment request
Paying a deposit by postTwo weeks of silenceCalling a serviceA timeout
Paying a deposit by postSending a second cheque "to be safe"Calling a serviceA blind retry: maybe a double charge
Paying a deposit by postChecking your bank statementCalling a serviceAsking the source of truth
Paying a deposit by postQuoting your reference numberCalling a serviceUsing an idempotency key

Where the analogy breaks: a lost letter is rare and slow. A busy system faces this "did my letter arrive?" question thousands of times per second.

Check your understanding

Q1/3Scenario

Match each incident to the fallacy behind it:

A cut cable between zones drops 2% of requests

A loop calling another region 40 times makes a page take 4 s

Autoscaling replaced servers, and a hardcoded IP stopped working

The payment provider changed a timeout without telling you

How confident are you?

Practice ladder

Step 1 · Trace

Trace booking bk_91 through a slow minute

Scenario Warm-up

Using Lina's new checkout (intent first, 8 s timeout, recovery job every minute):

19:06:00 Booking saves bk_91 as PENDING_PAYMENT and calls the provider with key bk_91.
19:06:08 No answer: timeout.
19:06:09 The provider finishes the charge (it was slow, not dead).
19:06:15 The guest, who sees a "Confirming…" page, taps "Back" and "Pay" again.
19:07:00 The recovery job runs.

Trace what happens at each step. How many times is the guest charged? What state does bk_91 end in?

Step 2 · Complete

Finish Lina's gap audit

Design Practice

Lina audits checkout step by step: (1) hold the room, (2) charge, (3) save the booking, (4) publish BookingConfirmed, (5) call the email API. She fills in what's left behind if the process crashes right after each step, and who fixes it. Two rows are missing.

Crash right after…	Left behind	Who fixes it
1. hold the room	A hold with no payment	Hold expires after 10 minutes
2. charge	___	___
3. save the booking	Booking saved, event published together via outbox	Outbox relay publishes it later
4. publish	Event out, no email yet	Email reader retries from the event
5. email	___	___

Fill in rows 2 and 5.

Step 3 · Solve

Did the hotel get the reservation?

Design Practice

After payment, StayHub sends each confirmed booking to the hotel's own system through a channel manager API: POST /reservations. On busy evenings about 1 in 500 calls times out after 10 s. Today the code logs a warning and moves on, and roughly once a week a guest arrives at a hotel that never received the reservation.

The channel manager supports GET /reservations?externalRef=bk_77, and accepts an externalRef on create (it rejects a second create with the same ref).

Design how StayHub should handle these timeouts. Name the states, who resolves the unknowns, and what tells you it's broken before a guest does.

Step 4 · Decide

Is nightly reconciliation worth a sprint?

Decision Stretch

The recovery job is live and Friday-style incidents have stopped. Karim: "Reconciliation is a sprint of work for two engineers. The recovery job already fixes stuck payments. Do we really need it?"

Facts: 12,000 bookings a day, about €200 average. Last quarter, three bugs on other teams created payment states the recovery job couldn't see (a refund that silently failed, a webhook handler that skipped a row, a manual charge from support).

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a junior engineer why 'just retry on error' is dangerous for a payment, and what to do instead.

0 words