Episode 5 · Charged twice on a train

What is an API?

Wednesday, 08:52: Inês tapped Pay once on StayHub's new iPhone app, on a train to Porto. Her bank shows two charges of €310, and she's the 14th guest charged twice since the app launched last week.

An API is the menu one program offers another: what you can ask for, how to ask, and what comes back. Here's how it works and why it must stay stable.

30 min foundationAPI as a contractREST resources & verbsHTTP methods & status codesIdempotencyAuthentication (who you are) vs authorization (what you may do)Evolving APIs without breaking clients

Problem
A web app, two mobile apps and partner travel agencies all need to search hotels and book rooms through your backend.
Why the simple system fails
Without a stable, explicit contract, every client guesses how to talk to the server; one field rename breaks old app versions, and a retried payment charges a guest twice.
Engineering concept
An API is a contract: named operations, input and output shapes, errors, auth and guarantees like idempotency, kept stable over time through versioning.
Technology
HTTP + JSON REST APIs, OpenAPI specs, bearer tokens, Idempotency-Key headers; GraphQL and WebSockets for other access patterns.
Trade-offs
REST is simple and cacheable but can over/under-fetch; GraphQL is flexible for clients but harder to cache and secure; WebSockets push in real time but hold connections open. Stability for clients costs flexibility for the server team.

The problem

2 min

Wednesday, 08:52. StayHub's first iPhone app launched last week.

Maya · Product manager
Inês booked a room in Porto from the train. One tap, two charges of €310. She's the 14th double charge since launch.
You · Engineer at StayHub
A double-tap bug?
Lina · Senior engineer, your mentor
The button disables after one tap. Guess what else could send it twice.

The big idea

4 min

One tap, two charges

The big idea

An API is a contract: what to send, what comes back, and what happens if you send it twice.

The app did the reasonable thing: it timed out and tried again. The gap is in the contract, which never said how to tell a retry from a new payment.

How it works

5 min

Our is the set of requests StayHub promises to answer, the shape of each, and what each answer means. The app never sees our database; it only knows the contract.

One payment, on a train

1
Request
The app sends POST /v1/payments with Inês's token and { bookingId: 90217 } as . POST means "create something"; GET means "read".
2
Work done
Our server charges €310 with the payment provider. Success.
3
Answer lost
We send 201 Created, a status code: 2xx worked, 4xx the client must change something, 5xx the server failed. The train is in a tunnel; it never arrives.
4
Timeout
After 10 s the app gives up. It can't tell "never arrived" from "answer lost".
5
Retry
The app sends the same request. To us it's a new payment: €310 again. POST isn't idempotent: doing it twice isn't the same as once.
6
The fix: a key
The app makes one per payment and repeats it on every retry. The server stores the result under it and replays it instead of charging again.

Watch a decision being made

4 min

Worked example

Lina makes payments safe to retry

14 guests were charged twice when the app retried after a timeout. Make retries safe, without making the app stop retrying.

1Replay the failure
In a test copy, make the server sleep 12 s after charging, so the 10 s timeout fires and the app retries. Two charges appear.
Why: You need a failure you can trigger on demand to prove the fix.
2Decide what makes two requests "the same"
Not the body: a guest may really pay twice for two bookings. The app creates one random key per tap on Pay and reuses it on every retry.
Why: Only the client knows which requests are retries of the same intention.

3Store and replay the result under the key

Look up the key (scoped to the signed-in user) before charging; save the response under it after.

const key = req.header("Idempotency-Key");
if (!key) return res.status(400).json({ error: "idempotency_key_required" });
const seen = await db.idempotency.find({ key, userId: req.user.id });
if (seen) return res.status(seen.status).json(seen.body); // replay, don't charge
const payment = await payments.charge({ bookingId, idempotencyKey: key });
await db.idempotency.insert({ key, userId: req.user.id, status: 201, body: payment });
res.status(201).json(payment);

Why: The server is the only place that knows whether the work already happened.

4Pass the key along
Send the same key to the payment provider, so even a crash between charging and saving can't charge twice.
Why: Our database and the provider are two systems; each must recognise the retry.

In real life

3 min

Two well-known APIs, two styles

Stripe: one careful REST API

Its public docs describe standard HTTP methods and an Idempotency-Key header on requests that create things
Changes ship as dated versions; each account stays on its version until it upgrades

Fits when correctness and stability for outside developers matter most.

GitHub: REST plus GraphQL

Publicly offers both a REST API and a GraphQL API
With GraphQL, a client asks for exactly the fields it needs in one request
Two contracts to document, secure and evolve

Fits when many clients need very different slices of a lot of data.

What goes wrong

3 min

Renaming or removing fields in place

Old apps and partners break instantly, and you can't force them to update.

Instead

Add new fields next to old ones; remove old ones on a timeline.

Retrying a POST without an idempotency key

The first charge may have succeeded; the retry charges again.

Instead

Require a key on create and charge requests; store and replay results.

Returning 200 OK with an error in the body

Apps, monitoring and retry logic read ; this looks like success.

Instead

Honest 4xx/5xx codes plus a structured error body.

Trade-offs

3 min

Option	Fits when	You pay	Main risk
REST over HTTP	Default, public and partner APIs	Several requests per screen	Over-fetching fields
GraphQL	Several in-house clients, varied data	Lost HTTP caching, query limits	One very expensive query
Additive change	Most changes	Old fields linger	Nobody removes them
New version (/v2)	A field's meaning changes	Everything maintained twice	A small team doing double work

Wednesday, 14:00. Cleaning up after the rename.

Karim · CTO
Sending both price and totalPrice is messy. Launch a clean /v2 and leave /v1 as it is.
Lina · Senior engineer, your mentor
Then every fix lands in two versions, for as long as v1.0 apps exist. One extra field costs almost nothing.
Karim · CTO
And in a year the API is full of old names nobody dares remove.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which endpoints change money or bookings and need an idempotency key (POST /payments yes, GET /hotels no).
What counts as a breaking change: add the new field next to the old one, remove the old one only when old app versions are gone.
Which status code each failure returns (400, 401, 403, 409, 500), because the apps decide what to show from it.

Tell Claude Code

14 of about 1,900 app payments were charged twice: the charge succeeded, the response was lost, the app's 10 s timeout fired and it retried POST /payments. Make it idempotent: the app sends an Idempotency-Key per payment, the server stores key and result, and a repeat returns the stored result without charging. Pass the key to the payment provider. Don't rename or remove response fields: app v1.0 depends on them. Plan first, then test one key sent twice gives one charge.

Check before you accept

The test: same key twice gives one charge and the same response; two different keys give two charges.
Keys are stored in the database, not one server's memory: StayHub has two app servers.
A diff of response shapes before and after: nothing renamed or removed (price is still there).

Red flags

It "fixes" double charges by turning off retries or lengthening the timeout.
It renames a field and updates only the web app, saying "clients are updated".
It uses GET for an action that changes something, like cancelling a booking.

Check yourself

2 min

Q1/3Scenario

On a flaky train connection, a guest taps 'Pay'. The request times out, the app retries automatically, and the guest is charged twice. What's the robust fix?

How confident are you?

Next episode: StayHub moves to a new server, and for a whole day some guests still land on the old one. Lina asks how a name like stayhub.com finds an address at all.

Finished?

Mark it complete to update your skill map.

Next: The internet, IP addresses and DNS

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Making a payment safe to retry

http

POST /v1/payments HTTP/1.1
Host: api.stayhub.com
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9…
Content-Type: application/json
Idempotency-Key: 7c1e4b2a-9d3f-4e8a-b1c6-2f0a5d7e9b31

{ "bookingId": 90217 }

HTTP/1.1 201 Created
Content-Type: application/json

{ "id": "pay_5521", "bookingId": 90217, "amountEur": 310.00, "status": "succeeded" }

A REST request and its response: method + path, headers, JSON body, then a status code and JSON back.

Worked example

Lina makes payments safe to retry

14 guests were charged twice when the iPhone app retried a payment after a timeout. Make retries safe, without making the app stop retrying.

1Replay the failure
In a test copy, make the server sleep 12 s after charging, so the app's 10 s timeout fires and it retries. Two charges appear.
Why: You need a failure you can trigger on demand to prove the fix works.
2Decide what makes two requests "the same"
Not the body: a guest may really pay twice for two bookings. The app creates one random key when the guest taps Pay, and reuses it for every retry of that tap.
Why: Only the client knows which requests are retries of the same intention.

3Store the result under the key

Before charging, look up the key (scoped to the signed-in user). After charging, save the status and response body under it.

const key = req.header("Idempotency-Key");
if (!key) return res.status(400).json({ error: "idempotency_key_required" });
const seen = await db.idempotency.find({ key, userId: req.user.id });
if (seen) return res.status(seen.status).json(seen.body);   // replay, don't charge
const payment = await payments.charge({ bookingId, idempotencyKey: key });
await db.idempotency.insert({ key, userId: req.user.id, status: 201, body: payment });
res.status(201).json(payment);

Why: The server is the only place that knows for sure whether the work already happened.

4Pass the key along
Send the same key to the payment provider, which supports it too. Then even if our server crashes between charging and saving, the provider won't charge twice.
Why: Our database and the provider are two systems; each needs to recognise the retry.
5Make the key required
Answer 400 to any payment without a key, and ship an app update that sends it. Keep the old app working by generating a key on the server only for requests from v1.0, which we can't change.
Why: An optional safety rule is a rule someone will forget.

Method	Typical use	Safe?	Idempotent?
GET	Read a resource or list	Yes	Yes
POST	Create something / trigger an action	No	No (unless you add a key)
PUT	Replace a resource entirely	No	Yes
PATCH	Change some fields	No	Not guaranteed
DELETE	Remove a resource	No	Yes

Safe = no side effects. Idempotent = doing it twice has the same effect as doing it once.

Status	Meaning	StayHub example
200 OK	Success, here's the data	GET /v1/hotels/42
201 Created	A new resource was created	POST /v1/bookings
400 Bad Request	Your input is invalid	Check-out before check-in
401 Unauthorized	Not signed in (no or invalid token)	Expired token
403 Forbidden	Signed in, but not allowed	Reading someone else's booking
404 Not Found	No such resource	Hotel 99999
409 Conflict	Conflicts with the current state	Room just booked by someone else
429 Too Many Requests	Slow down	A script looping on search
500 / 503	Server bug / temporarily unavailable	Database down; retrying later may help

4xx: the client should change something. 5xx: the server failed; retrying may help.

Interactive simulation

Visual diagram

Web app

Mobile app (v1 & v2)

Partner agency

API: /v1/…

Booking service

Database

Payment provider

Click a component

One Booking API, many clients. Walk through a search, a booking and a safely retried payment.

Analogy

Hotel room serviceAn API

Hotel room serviceThe printed menuAn APIThe API documentation
Hotel room service"Room 312, one club sandwich"An APIA request with login and body
Hotel room service"Coming in 20 minutes" or "Sold out"An APIA response with a status code
Hotel room service"Is this order #A7 again?"An APIAn idempotency key stops duplicates
Hotel room serviceHonouring the old menuAn APIKeeping old API versions working

Where the analogy breaks: a waiter can notice a confused guest and help. An API answers exactly what it was asked, millions of times a day, so every rule has to be written down in advance.

Check your understanding

Q1/3Reasoning

GET /bookings/42/cancel is an acceptable design as long as the endpoint checks authentication.

How confident are you?

Practice ladder

Step 1 · Trace

Trace four payment requests

Scenario Warm-up

StayHub's POST /v1/payments now stores results under the Idempotency-Key, scoped to the signed-in user. Trace these four requests, in order, and say for each: does the card get charged, and what status code comes back?

Inês sends key K1 for booking 90217. The charge succeeds, but the response is lost.
Inês's app retries with key K1.
Inês books a second room (booking 90240) and pays with key K2.
A buggy partner app sends a payment with no key at all.

Step 2 · Complete

Finish Lina's status-code review

Code Practice

Lina is reviewing GET /v1/bookings/:id. Today it returns 200 in every case, with an error field in the body when something is wrong. She wrote the table and left gaps:

Situation	Status today	Should be
Guest's own booking found	200	200
No token, or token expired	200 + error	___
Signed in, booking belongs to another guest	200 + error	___
Booking id doesn't exist	200 + error	___
Database is down	200 + error	___

Fill in the four gaps and explain, in one line each, what the app should do in response.

Step 3 · Solve

Design the cancel request

Design Practice

Maya wants guests to cancel a booking from the app, free up to 48 hours before check-in. The first draft is:

GET /v1/bookings/90217/cancel, which cancels and refunds, and returns 200 with { "ok": true } or 200 with { "ok": false }.

Redesign this request: the method and path, what the app sends, the possible responses with status codes, and how a retry on a bad connection stays safe. Remember that the app may be on a train.

Step 4 · Decide

When can we drop "price"?

Decision Stretch

Two months after the rename, the API still sends both price and totalPrice. Usage of the iPhone app:

v1.0 (reads price): 6% of app users, about 30 bookings a week
v1.1 and later (read totalPrice): 94%

Karim: "Remove price next week, 6% is nothing." Omar: "Keep it forever, it's one field."

Write a short decision: "We chose X over Y because…, accepting the risk of…", and include what you'd do before removing anything.

Explain it yourself

Explain to a teammate what an API contract is, and why POST /payments needs an idempotency key while GET /hotels/42 doesn't.

0 words