Episode 44 · Charged twice in a tunnel

Idempotency and safe retries

Saturday, 18:40: since Thursday's app release, 212 guests have been charged twice, the same €360 a few seconds apart. The new app retries payments when the network drops, exactly as it was designed to.

Networks fail, so you'll always need to "try again". Here's how to try again without charging a guest twice.

30 min intermediateIdempotencyIdempotency keys & dedupe tablesWhen a retry is safeAt-least-once deliveryPartial failures

Problem
A guest taps 'Pay €480'. The request times out. Did the charge happen?
Why the simple system fails
Not retrying loses bookings; retrying blindly charges the guest twice. The network cannot tell you which one happened.
Engineering concept
Make operations idempotent: the same request applied twice has the same effect as once. Then retries become safe.
Technology
Idempotency-Key headers, a dedupe table with a unique constraint, stored responses, idempotent consumers on at-least-once queues.
Trade-offs
Extra storage and a write on every request, key retention windows, and careful handling of concurrent duplicates and changed payloads.

The problem

2 min

Saturday, 18:40. Maya forwards a bank statement: €360, €360.

Maya · Product manager
212 guests charged twice since Thursday's app release. One tap on Pay, two charges.
Omar · On-call SRE
v5.2 retries Pay after a 10 s timeout, up to 3 times. Trains, tunnels, lifts.
Lina · Senior engineer, your mentor
What does our server see on the retry?

The big idea

4 min

One tap, two charges

The big idea

At-least-once delivery + idempotent handling = the effect of exactly once.

Turning retries off doesn't help: guests just tap Pay again by hand. Retries come from users, phones, proxies, queues and webhooks. So make them safe, not rare: give each intent a name the server can recognise.

How it works

5 min

One checkout, one key

1
Prefer "set" over "add"
An operation is if doing it twice has the same effect as once. "Set status to cancelled" is; "charge €360" isn't. Where you can write "set X to Y", retries are already safe.
2
Create the key where the intent is born
An is a unique id for one intent ("pay for this cart"). The app creates it when the guest taps Pay and stores it with the cart, so it survives app restarts.
3
Send it on every attempt
The first request and every retry carry Idempotency-Key: 7c1e…. A new key per attempt dedupes nothing.
4
The server claims it first
The stores keys it has handled, with the answer it gave. The handler starts with INSERT … ON CONFLICT DO NOTHING on the key's primary key: only one request wins, decided by the database.
5
Repeats get the same answer
Same key, done: replay the stored response. Still running: 409, retry soon. Same key, different body: 422, a client bug.
6
Pass it down, save together
Charge the provider with pay-7c1e…, so it dedupes too. Save the payment and the stored response in one transaction. Purge keys after 24 h.

Watch a decision being made

4 min

Worked example

You design the fix; Lina checks it

Stop the app's retries from charging twice, without turning retries off. Up to 3 retries; 7,000 app payments a day.

1Find where the intent is born
"Pay for this cart" is born when the guest taps Pay. The app creates a UUID there and saves it with the cart.
Why: A key created per request on our server dies with the retry. It must outlive restarts and tunnels.
2Claim the key atomically
First thing on the server: insert the key; only a request whose insert returns a row may continue.
sql
```
INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;
```
Why: Two copies can arrive 40 ms apart. A unique constraint picks exactly one winner; a SELECT check can't.
3Decide what repeats get
Completed: replay the stored response. In progress: 409. Different body hash: 422.
Why: Replaying gives the app the real result. The hash catches a key reused for a different amount.
4Pass it down and save together
Charge with provider key pay-7c1e…. Then, in one transaction: insert the payment, mark the key completed, store the response.
Why: If the server crashes after charging, the provider recognises the key on the retry and doesn't charge again.

In real life

3 min

Remember the request versus declare the result

Stripe: idempotency keys

Publicly documents an Idempotency-Key header on its create requests
Same key again returns the original result, for a limited window (documented as 24 hours)
Works for actions that can't be repeated, like charging a card

Fits actions that add something: a charge, an order, an email.

Kubernetes: declare the desired state

You describe the end state ("3 copies of this app") and apply it
Applying it twice changes nothing: it's already true
No keys to store, but only for actions you can write as "make it so"

Fits updates expressed as "set X to Y".

What goes wrong

3 min

Check, then act

Two concurrent duplicates both see "not seen" and both charge.

Instead

Claim first with INSERT … ON CONFLICT DO NOTHING; act only if your insert won.

A new key on each retry

Every attempt looks like a new intent, so nothing is deduped.

Instead

One key per checkout, stored with the cart, reused for every attempt.

Recording "processed" separately from the effect

A crash between them makes the retry repeat the effect.

Instead

Commit the dedupe row and the business change in one transaction, and pass a key to the provider.

Trade-offs

3 min

Option	Fits when	You pay
Naturally idempotent ("set", upserts)	You can write it as "set X to Y"	Not possible for charges, emails, counters
Idempotency key + dedupe table	Money, bookings, outside side effects	A write per request; clients must keep keys right
Unique constraint on business fields	A short-term stopgap	Guesses intent; blocks legitimate repeats

Saturday, 19:30. The war room.

Karim · CTO
Keys need an app release; stores take days. Tonight: a unique constraint on (guest, hotel, dates). Ships in an hour.
Lina · Senior engineer, your mentor
That also blocks guests booking a second room for the same nights, about 300 a day.
Karim · CTO
300 annoyed guests beat 100 double charges a day.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Where the key is born: in the app, on tap, saved with the cart, reused on every retry. Not on the server, not per request.
What a repeat gets back: the stored response, 409 while in progress, 422 for the same key with a different amount.
Which operations need keys: payments and bookings yes; absolute writes like "set status to cancelled" are already safe.

Tell Claude Code

212 StayHub guests were charged twice because app v5.2 retries a timed-out Pay as a fresh request. Plan first. The app creates one Idempotency-Key per checkout on tap, stores it with the cart, reuses it on every retry. Server: claim the key first with INSERT ON CONFLICT DO NOTHING (primary key); store a body hash; replay, 409 or 422 for repeats. Forward pay-<key> to the provider. Save payment and response in one transaction. Purge after 24 h. Test two identical requests 40 ms apart.

Check before you accept

The handler's first database call is the atomic INSERT … ON CONFLICT, not a SELECT.
Two requests 40 ms apart produce exactly one charge in the provider's test dashboard.
The same key with a different amount returns 422, not a second charge.

Red flags

It checks with a SELECT, charges, then inserts the key.
It generates a fresh UUID for every attempt.
It "fixes" duplicates by turning retries off in the app.

Check yourself

2 min

Q1/3Scenario

Your checkout calls POST /payments without an idempotency key. After 10 s it times out. What do you actually know?

How confident are you?

Next episode: the reviews service gets slow on a Friday night, and within four minutes the whole of StayHub, booking included, stops answering.

Finished?

Mark it complete to update your skill map.

Next: Timeouts, backoff and circuit breakers

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Let the database decide who wins

The contract. The client sends one key per intent. The server promises: the first request with a key runs; a later request with the same key and the same body gets the same response; the same key with a different body is refused.

http

POST /v1/bookings/bk_8812/pay HTTP/1.1
Host: api.stayhub.com
Content-Type: application/json
Idempotency-Key: 7c1e9b0a-4f2d-4d43-9a51-0f6f2b3e8d11

{ "amount": 36000, "currency": "EUR" }

HTTP/1.1 201 Created
Idempotent-Replayed: true

{ "paymentId": "pay_51", "status": "succeeded" }

A retry: same key, same body, original result. `Idempotent-Replayed` helps the app log what happened.

Worked example

You design the fix; Lina checks it

Stop the app's retries from charging twice, without turning retries off. The app sends Pay up to 3 times; 7,000 app payments a day.

1Find where the intent is born
The intent is "pay for this cart", born when the guest taps Pay. The app creates a UUID there and saves it with the cart.
Why: A key created per request (on our server) or per screen dies with the retry. It must outlive restarts and tunnels.
2Claim the key atomically
First thing on the server: INSERT … ON CONFLICT (key) DO NOTHING RETURNING key. Only a request whose insert returns a row may continue.
Why: Two copies can arrive 40 ms apart. A unique constraint lets the database pick exactly one winner; a SELECT check can't.
3Decide what repeats get
Same key, completed → replay the stored response. Same key, in progress → 409, retry soon. Same key, different body hash → 422.
Why: Replaying gives the app the real result. The hash catches a client bug that reuses a key for a different amount.
4Pass it down
Charge the provider with key pay-7c1e…, derived from the client's key.
Why: If our server crashes after charging and the app retries, the provider recognises the key and doesn't charge again.
5Save result and response together
In one transaction: insert the payment row, set the key's status to completed and store the response. Purge keys after 24 h.
Why: If they were separate, a crash between them would leave a charge with an "in progress" key, and the next retry would be stuck or repeat the work.

async function pay(req: Request, body: PaymentInput) {
  const key = req.headers.get("Idempotency-Key");
  if (!key) return json(400, { error: "Idempotency-Key required" });
  const hash = sha256(req.method + req.url + JSON.stringify(body));

  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, request_hash, status)
     VALUES ($1, $2, 'in_progress') ON CONFLICT (key) DO NOTHING RETURNING key`, [key, hash]);

  if (claimed.rowCount === 0) {
    const row = await db.one("SELECT * FROM idempotency_keys WHERE key = $1", [key]);
    if (row.request_hash !== hash) return json(422, { error: "Key reused with a different request" });
    if (row.status === "completed") return json(row.response_code, row.response_body, { "Idempotent-Replayed": "true" });
    return json(409, { error: "Still in progress, retry shortly" });
  }

  const charge = await psp.charges.create(body, { idempotencyKey: `pay-${key}` });
  const response = { paymentId: charge.id, status: charge.status };
  await db.tx(async (t) => {
    await t.query("INSERT INTO payments (booking_id, charge_id) VALUES ($1, $2)", [body.bookingId, charge.id]);
    await t.query("UPDATE idempotency_keys SET status = 'completed', response_code = 201, response_body = $2 WHERE key = $1", [key, response]);
  });
  return json(201, response);
}

Claim, branch on what you find, do the work once, save result and response together.

Interactive simulation

Visual diagram

Guest app

Booking API

idempotency_keys

Bookings DB

Payment provider

Booking events

Click a component

A payment request that times out, is retried, and is answered from the dedupe table. Start the walkthrough.

Analogy

Paying rent by chequeCalling a payment API

Paying rent by chequeMailing a cheque, hearing nothingCalling a payment APIA request that times out
Paying rent by chequeThe cheque number, #1042Calling a payment APIThe idempotency key
Paying rent by chequeSending a copy with the same numberCalling a payment APIRetrying with the same key
Paying rent by chequeThe bank's list of cashed numbersCalling a payment APIThe server's dedupe table
Paying rent by chequeCopy ignored, paid only onceCalling a payment APIOne charge, original answer returned

Where the analogy breaks: you write a cheque number by hand once. In code, the key must be created once per intent, where the click happens, not once per attempt, or every retry looks brand new.

Check your understanding

Q1/4Reasoning

Because HTTP defines PUT as idempotent, every PUT endpoint in your API is automatically safe to retry.

How confident are you?

Practice ladder

Step 1 · Trace

Trace four requests with one key

Scenario Warm-up

Using Lina's pay handler, trace these requests. All use Idempotency-Key: 7c1e….

Time	Body	Note
18:00:00.000	amount 36000	First tap
18:00:00.040	amount 36000	Double tap; the first is still charging
18:00:12.000	amount 36000	App retry after a timeout; the first finished at 18:00:00.900
18:05:00.000	amount 18000	A bug reuses the key for a different booking

For each, give the HTTP status and whether the card is charged.

Step 2 · Complete

Put the server's steps in order

Design Practice

Lina wrote the steps for a request with a key the server has never seen, then shuffled them. Put them in order and say, for the two steps marked ★, what goes wrong if they're swapped.

A. Charge the provider with key pay-<key>
B. ★ Claim the key with INSERT … ON CONFLICT DO NOTHING
C. Read the Idempotency-Key header; refuse with 400 if missing
D. In one transaction: save the payment, mark the key completed, store the response
E. Compute the request hash
F. ★ Return 201 to the app

Step 3 · Solve

Three confirmation emails

Debugging Practice

The payment provider sends webhooks to POST /webhooks/payments, for example { "id": "evt_93", "type": "payment.succeeded", "bookingId": "bk_8812" }. It retries until it gets a 2xx, sometimes several times within a second.

The handler marks the booking paid and sends the confirmation email. Some guests get three confirmation emails, and once a refund webhook arrived before the success webhook and left a refunded booking marked "paid".

Make the webhook handler safe against repeats and order problems.

Step 4 · Decide

Require keys from hotel partners?

Decision Stretch

Hotels' own systems call StayHub's partner API about 4 million times a day to update prices and rooms (POST /rates, POST /availability). Some partners retry blindly on timeouts. Last month a retried POST /availability { "add": 2 } gave a hotel 2 phantom rooms, which were then sold.

Option A: require Idempotency-Key on every partner POST within 90 days (about 2 GB of keys for a 24 h window). Option B: change the API to absolute values (PUT /availability { "roomsLeft": 5 }) and deprecate add.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a junior engineer why 'just retry on timeout' can charge a guest twice, and how an idempotency key prevents it.

0 words