Episode 22 · One payment, three confirmations

Webhooks: receiving and sending events over HTTP

Monday, 11:20: StayHub's first 3-D Secure test payment is confirmed by the provider's webhook, three times. The guest got three confirmation emails, and Lina says: "You lead this one."

Instead of asking "anything new?" every few seconds, you let other systems call you when something happens. Here's how to do it safely.

30 min intermediateReceiving webhooks: signatures, replay and orderingSending webhooks reliablyIdempotencyAt-least-once deliveryBackground jobs, queues and workersJob retries, backoff and dead-letter queues

Problem
Your app must learn about events that happen in someone else's system (a payment succeeded after 3-D Secure, a booking arrived from a partner site) and tell partners about yours.
Why the simple system fails
Polling every few seconds is slow and wasteful. A naive webhook endpoint trusts anyone who knows the URL, processes duplicates twice, times out doing slow work, and applies events in the wrong order.
Engineering concept
Treat each webhook as an untrusted, possibly duplicated, possibly late message: verify its signature, record it, acknowledge fast, process it idempotently in the background, and reconcile against the source of truth.
Technology
HTTPS POST with a JSON body, HMAC signatures with timestamps, an inbox table with a unique event id, background workers, retry schedules with backoff on the sending side, and replay/reconciliation tools.
Trade-offs
Near-real-time integration with little load, in exchange for at-least-once delivery, no ordering guarantees, a public endpoint to secure, and a dependency on the other side's retry behaviour.

The problem

2 min

Monday, 11:20. The first 3-D Secure test payment.

Omar · On-call SRE
Test booking bk_81723: the provider sent payment_intent.succeeded 3 times.
Maya · Product manager
And the guest got 3 confirmation emails.
Lina · Senior engineer, your mentor
You lead this one. I'll only check your answers. Start with a guess.

The big idea

4 min

They call us

The big idea

A is the provider calling us, and every delivery may be forged, repeated, late or out of order, or never arrive.

So the handler can't be "do the work, then answer". It must check who sent it, remember what it already saw, answer fast, and do the real work later, safely, however many times it arrives.

How it works

5 min

With 3-D Secure, the payment finishes after checkout returned "pending". So we give the provider a URL, and it calls us: an API call in reverse. Here's a receiver built for all four dangers.

Verify, record, acknowledge, process, reconcile

1
Verify
Recompute the : a code made from the raw body, a timestamp and a secret only we and the provider know. Wrong, or older than 5 minutes? Answer 400 and store nothing.
2
Record once
Insert the event into an inbox table with a unique event id. A repeated delivery inserts nothing.
3
Acknowledge
Enqueue a job and return 200 in about 40 ms. Nothing slow happens here, so the provider never times out.
4
Process
A worker confirms the booking with a guarded update: it only moves pending → confirmed. A duplicate or stale event changes 0 rows, and the email only follows a real change. That makes it idempotent: safe to run twice.
5
Reconcile
Reconciliation: every hour, ask the provider for recent payments and confirm any booking still pending. A lost webhook becomes a delay, not a lost booking.

Watch a decision being made

4 min

Worked example

You rebuild the payment webhook, Lina checks

Replace the 12-second handler so a payment confirms its booking exactly once, even when the delivery is forged, repeated, late or out of order.

1Verify before anything
Read the raw body, recompute the signature over timestamp + body, compare in constant time, reject anything older than 5 minutes.
ts
```
const raw = await req.text();   // raw bytes, not re-serialized JSON
const expected = createHmac("sha256", SECRET).update(ts + "." + raw).digest("hex");
if (!safeEqual(expected, sig) || Math.abs(Date.now() / 1000 - ts) > 300) return status(400);
```
Why: Re-serialized JSON changes the bytes, the signature never matches, and teams switch checks off "for now". The timestamp blocks a .
2Record and dedupe
Insert into webhook_events with UNIQUE(provider, event_id) and ON CONFLICT DO NOTHING. Enqueue a job only if the row is new.
Why: Every repeated delivery becomes a no-op, and the table is an audit trail we can replay after a bug fix.
3Acknowledge fast, even for events we ignore
Return 200 in about 40 ms. Valid event types we don't handle get 200 too. Only a bad signature gets a 4xx.
Why: A slow answer causes retries; an error on an ignored event makes the sender retry for days.
4Process with a guard
The worker re-reads the payment from the provider's API, then runs UPDATE bookings SET status = 'confirmed' WHERE id = $1 AND status = 'pending'. One row changed? Enqueue the email.
Why: Re-fetching makes order irrelevant; the guard turns duplicates into 0-row updates, so the email never repeats.

In real life

3 min

Two well-known senders, two retry policies

Stripe: retries automatically

Documents automatic retries with backoff for up to three days
Signs each delivery and includes a timestamp
Receivers must expect duplicates

Fits events that must not be missed, like payments.

GitHub: no automatic redelivery

Documents that failed deliveries aren't retried automatically
Offers manual redelivery from settings or its API
Also signs each payload with a shared secret

Fits events where a missed one can be caught up later.

What goes wrong

3 min

Returning 500 for events you don't handle

The sender retries for days and may disable the endpoint, taking the events you need with it.

Instead

2xx for valid but irrelevant types; 4xx only for bad signatures.

Doing the real work inside the handler

It exceeds the sender's timeout, the sender retries, and the work runs again.

Instead

Verify, record, enqueue, return 200. Process in a worker.

Verifying the signature on re-serialized JSON

The bytes change, it never matches, and verification gets switched off.

Instead

Verify on the raw request bytes, then parse.

Trade-offs

3 min

Option	Fits when	You pay
Webhooks (push)	You must react within seconds	A public endpoint to secure; duplicates, disorder, misses
Polling (pull)	Minutes of delay are fine, or as a safety net	Delay up to the interval; wasted calls, rate limits
Shared event stream	Internal services, high volume, order matters	Both sides must run or reach the platform

Thursday, 14:00. The endpoint is back on.

Karim · CTO
The provider retries for three days. Do we really need a reconciliation job too? Another thing to build and watch.
Omar · On-call SRE
Three days of retries didn't save us this week. And a worker bug loses events even with a perfect endpoint.
Karim · CTO
It's an API call every hour, against their rate limits.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

That no webhook is trusted until its signature and timestamp are checked. A "payment succeeded" from anyone else is a free stay.
What the endpoint does before answering: verify, store by event id, queue the work, reply 200 well under the provider's 10 s timeout.
What each status code means: 200 for any valid event, even types you ignore; errors only for bad signatures or your own temporary failures.

Tell Claude Code

Our payment webhook handler does the work before answering, about 12 s; the provider times out at 10 s and retries, so bk_81723 got 3 emails. Plan first. Rework /webhooks/payments: verify the signature on the raw body, reject timestamps older than 5 minutes, insert into an inbox table with a unique event id, enqueue a job, return 200 in under a second. The job confirms only pending bookings. Return 200 for unhandled types. Test: forged signature, same event 3 times, unknown type.

Check before you accept

A test with a wrong signature: rejected, nothing stored.
A test sending the same event id 3 times: one confirmation, one email, one inbox row.
An unknown event type gets 200, and a reconciliation job compares our bookings with the provider's payments.

Red flags

The endpoint is "protected" only by a secret-looking URL, or verifies re-serialized JSON.
It returns 500 (or 400) for event types it doesn't recognise.
It keeps the slow work before replying and suggests asking the provider for a longer timeout.

Check yourself

2 min

Q1/3Scenario

Your /webhooks/payments endpoint is only "protected" by an unguessable URL. What can go wrong?

How confident are you?

Next episode: Ana changed her email, yet 3 of her 14 booking confirmations went to the old address, and finance finds 41 bookings that check out before they check in.

Finished?

Mark it complete to update your skill map.

Next: SQL and relations: modelling a hotel in tables

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Verify, record, acknowledge, process

Here's what a delivery from the provider looks like. Header names vary by provider; the structure doesn't:

http

POST /webhooks/payments HTTP/1.1
Host: api.stayhub.com
Content-Type: application/json
Webhook-Id: evt_3Nf8a2
Webhook-Timestamp: 1790000000
Webhook-Signature: v1=5d0c8b6f0a…e91

{"id":"evt_3Nf8a2","type":"payment_intent.succeeded","created":1790000000,
 "data":{"object":{"id":"pi_77","amount":24000,"currency":"eur","metadata":{"bookingId":"bk_81723"}}}}

Event id for dedupe, timestamp against replays, signature over timestamp + raw body.

Worked example

You rebuild the payment webhook, Lina checks

Replace the quick handler so a payment webhook confirms its booking exactly once, even when it's forged, repeated, late, out of order, or lost during an outage.

1Verify before anything

Read the raw body, recompute HMAC-SHA256(secret, timestamp + "." + raw), compare in constant time, and reject timestamps older than 5 minutes.

const raw = await req.text();                     // raw bytes, not re-serialized JSON
const expected = createHmac("sha256", SECRET).update(ts + "." + raw).digest("hex");
if (!safeEqual(expected, sig) || Math.abs(Date.now() / 1000 - ts) > 300) {
  return new Response("rejected", { status: 400 });
}

Why: Lina's check: re-serialized JSON changes the bytes, so the signature never matches and teams "temporarily" switch verification off. The timestamp blocks a .

2Record and dedupe
Insert into webhook_events with UNIQUE(provider, event_id), ON CONFLICT DO NOTHING. Enqueue a process_psp_event job only if the row was new.
Why: The unique constraint turns every repeated delivery into a no-op, and the table is an audit trail we can replay from after a bug fix.
3Acknowledge fast
Return 200 right away, about 40 ms. Also return 200 for valid event types we don't handle. Only a bad signature gets a 4xx.
Why: Slow answers cause retries, and errors on events we ignore make the sender retry for days, then possibly disable the endpoint.
4Process with a guard
The worker re-reads the payment from the provider's API (the source of truth), then runs UPDATE bookings SET status = 'confirmed' WHERE id = $1 AND status = 'pending'. If 1 row changed, enqueue the confirmation email.
Why: Re-fetching makes order irrelevant; the guard makes duplicates and stale events change 0 rows, and the email only follows a real change.
5Reconcile on a schedule
Every hour, list the provider's payments updated since the last run and confirm any booking still pending whose payment succeeded.
Why: Senders give up eventually. Reconciliation turns a lost webhook into a delay of at most an hour instead of a paid guest with no booking.

Concern	Receiver does	Sender does
Authenticity	Verify HMAC on the raw body, constant-time compare	Sign with a per-subscriber secret, support rotation
Replay attacks	Reject timestamps outside ~5 min; dedupe event ids	Include the timestamp in the signed content
Duplicates	UNIQUE(event_id) inbox; idempotent handlers	Keep the same event id across retries
Timeouts	Acknowledge in < 1 s, process in the background	Short timeout; retry with backoff and jitter
Ordering	State guards, versions, or re-fetch	Include a version; don't promise order
Missed events	Reconciliation job against the sender's API	Retry for days; delivery log; replay button

Webhook reliability is a contract with two sides.

Interactive simulation

Visual diagram

Payment provider

Webhook endpoint

Inbox table

Job queue

Event worker

Bookings DB

Webhook dispatcher

Channel manager

Click a component

Receiving a payment webhook (left to right) and sending availability webhooks to channel managers (bottom).

Analogy

Letters from your bankWebhooks

Letters from your bankGiving the bank your addressWebhooksRegistering your webhook URL
Letters from your bankA wax seal only the bank can makeWebhooksThe HMAC signature
Letters from your bankThe courier comes back if you're outWebhooksRetries, so duplicates happen
Letters from your bankSign now, read laterWebhooksReply OK fast, process in background
Letters from your bankCalling the bank to check the balanceWebhooksRe-fetching the real state from their API

Where the analogy breaks: a lost letter is usually gone for good. With webhooks, a nightly job that compares your data with the sender's can find and fix what was missed.

Check your understanding

Q1/3Reasoning

If a provider sent payment.succeeded before charge.refunded, you can rely on receiving them in that order.

How confident are you?

Practice ladder

Step 1 · Trace

Trace four deliveries

Scenario Warm-up

StayHub's new receiver: verify signature + 5-minute timestamp window → insert into webhook_events (UNIQUE(provider, event_id)) → enqueue only if new → return 200. The worker runs UPDATE bookings SET status='confirmed' WHERE id=$1 AND status='pending' and emails only if 1 row changed.

Booking bk_9020 is pending. These arrive in order:

evt_A payment_intent.succeeded for bk_9020, valid signature, fresh timestamp.
evt_A again (a provider retry), valid, fresh.
evt_B charge.refunded for bk_9020, valid, fresh. (The worker sets status to refunded.)
A copy of evt_A captured by an attacker, re-sent 2 hours later with its original headers.

For each, give the HTTP response, whether a job is enqueued, and the booking status afterwards. How many emails go out?

Step 2 · Complete

Put the receiver in order

Code Practice

A teammate wrote these steps for the channel-manager booking webhook (partners send us bookings made on their sites), but in the wrong order, and one step is missing. Put them in the right order and add the missing one.

(a) Return 200
(b) Parse the JSON body
(c) Insert into webhook_events with ON CONFLICT (provider, event_id) DO NOTHING
(d) Create the booking in the database and send the confirmation email
(e) Verify the HMAC signature on the raw body
(f) Enqueue process_cm_event if the row was new

Step 3 · Solve

Design the room.sold sender

Design Practice

Now StayHub is the sender. When a room sells, we must tell 12 channel managers within seconds, so they stop selling it. One partner is often slow (8–15 s per request); another is down for hours about once a month. Two partners complained they once received "room available" after "room sold out".

Design the sending side: how events are created, how they're delivered, retries, isolation between partners, ordering, and what partners need from us to stay safe.

Step 4 · Decide

Webhook or polling for a small hotel system?

Decision Stretch

A small property-management system (used by 9 of our hotels) offers two ways to learn about bookings made at the front desk: webhooks (no signature, no retries, "best effort") or a list endpoint GET /bookings?updated_since=… (rate limit: 60 calls a minute).

Front-desk bookings must reach StayHub fast enough that we don't sell the same room online. Today, about 30 front-desk bookings happen per hotel per day.

Write a short decision in this form: "We chose X over Y because…, accepting the risk of…". Say how you'd make up for the weaknesses of your choice.

Explain it yourself

Walk through what your payment webhook endpoint does from the moment the request arrives until the booking is confirmed, and why each step is there.

0 words