Episode 32 · Checkout waits for everyone

Event-driven architecture

Wednesday, 16:05: checkout takes 2.8 seconds at p95, and about 50 bookings a day fail because a hotel's channel manager times out, after the guest's card was already charged.

Instead of telling every other part what to do, a service announces what happened, and whoever cares reacts on their own.

30 min intermediateEvent-driven architectureEvents vs commandsChoreography vs orchestrationDesigning for eventual consistencyTransactional outboxIdempotency

Problem
Confirming a booking must trigger email, loyalty points, invoicing, analytics, fraud checks and partner sync, and the list keeps growing.
Why the simple system fails
If the booking code calls each of them synchronously, it is slow, fails when any of them fails, and must change every time a new team wants to react.
Engineering concept
The booking publishes a fact ('BookingConfirmed'). Interested parts subscribe and react on their own time. Producers don't know consumers; consistency between them becomes eventual.
Technology
Domain events, pub/sub brokers and logs (Kafka-style topics, managed queues), transactional outbox, idempotent consumers, schema registries, correlation ids and tracing.
Trade-offs
You gain decoupling, resilience and extensibility; you pay with eventual consistency, harder debugging, duplicate and out-of-order delivery, and contracts that are harder to change.

The problem

2 min

Wednesday, 16:05. Maya brings the checkout numbers.

Maya · Product manager
Checkout p95 is 2.8 s. And about 50 bookings a day fail after the guest's card was charged.
Omar · On-call SRE
All channel-manager timeouts. Booking calls the hotel's system and waits for it.
You · Engineer at StayHub
So we refund 50 guests a day because a hotel's system is slow?

The big idea

4 min

Ask, or announce?

The big idea

Keep the decision synchronous, then announce the fact as an and let everyone else react on their own schedule.

A command is the manager telling housekeeping "clean room 214 now": one recipient, who can say no. An event is the front desk announcing on the radio "guest checked out of 214": housekeeping, minibar and billing each act, and the desk waits for no one.

How it works

5 min

Only two steps decide whether a booking exists: holding the room and charging the card. Everything else is a reaction. Booking publishes one fact; others subscribe: .

A booking, event-driven

1
Command: decide
A command is a request to one handler, which can refuse it. ConfirmBooking holds the room and charges the card. The guest waits for this, about 0.5 s.
2
Record the fact
In the same transaction, Booking writes the booking and a BookingConfirmed row in the : a table where the event is saved with the change, then published by a relay.
3
Publish
A relay publishes outbox rows to a queue. After a crash it may publish one twice ().
4
Fan out
Email, loyalty, invoicing and channel sync each get every event and react on their own.
5
Catch up
If the hotel's system is down, channel sync retries later. Other parts are briefly behind: , catching up a little later.

Events flip the dependency: before, Booking depended on every consumer; now they depend on Booking's event, and a new team can subscribe without changing Booking.

Watch a decision being made

4 min

Worked example

You redesign checkout (Lina reviews)

Six steps in a row: p95 2.8 s, ~50 charged-but-failed bookings a day. The guest should wait only for what matters, and nothing may get lost.

1Split the decision from the reactions
Hold (30 ms) and charge (400 ms) stay synchronous. Invoice, email, loyalty and channel sync become consumers.
Why: The guest needs a yes or no, not the invoice. p95 goes from 2.8 s to about 0.5 s.
2Record the fact with the outbox
The booking and the BookingConfirmed outbox row commit in one transaction; a relay publishes unsent rows.
ts
```
await db.transaction(async (tx) => {
  await tx.insert(bookings).values(booking);
  await tx.insert(outbox).values({ type: "BookingConfirmed", eventId, payload });
});
```
Why: Commit-then-publish loses events when the process dies in between. One transaction means "saved" and "announced" can't disagree.
3Make every consumer idempotent
Each consumer records eventId in processed_events in the same transaction as its work and skips duplicates.
Why: Delivery is at least once. Without this, a crash means two emails or double points.
4Give the risky step an owner, and measure the lag
Channel sync gets a small orchestrator: retries for up to 6 hours, then alerts Hotel Partners. Alert if the oldest unsent outbox row is older than 60 s.
Why: Pure listeners hide where a step is stuck, and async work fails quietly. If you don't measure the delay, hotels tell you.

In real life

3 min

A central event log, or jobs inside the app

LinkedIn: a company-wide event log

Created Kafka and has publicly described it as a central log between many systems
Needs a team to run the brokers and manage event schemas

Fits many separate systems reacting to the same events.

37signals: one app, background jobs

Publicly advocates a single "majestic monolith" (Basecamp, HEY)
Runs reactions as background jobs; released Solid Queue, a database-backed queue

Fits when producer and consumers live in one app.

What goes wrong

3 min

Save to the database, then publish

A : a crash between them loses the event.

Instead

The transactional outbox.

Making everything asynchronous

"Never oversell" can't be eventual; the guest needs an answer now.

Instead

Decision synchronous, reactions asynchronous.

Commands disguised as events ("SendWelcomeEmail")

The producer still decides what others do, with a queue in between.

Instead

Publish facts ("GuestRegistered"); consumers decide.

Trade-offs

3 min

Style	Fits when	You pay
Synchronous calls	The caller needs the result to continue	Latency adds up; availability multiplies down
Events, choreography	Fan-out: notifications, indexing, loyalty	Eventual consistency; flows hard to see
Orchestration	Multi-step processes with retries and deadlines	One more critical component

Monday, 10:00. Choosing where events travel.

Karim · CTO
If we do events, let's do them properly: a managed Kafka cluster.
Lina · Senior engineer, your mentor
All our consumers live in one app. An outbox and our workers do it with nothing new to run.
Karim · CTO
And when we extract services next year, we redo everything.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which steps decide whether a booking exists (hold, charge: synchronous) and which are reactions (invoice, email, loyalty, channel sync).
What the guest sees while reactions catch up: "Booking confirmed, your email will arrive in a minute", read from Booking itself.
Which processes need an owner with state and a deadline (channel sync: 6 hours, then alert), and the lag you alert on (60 s).

Tell Claude Code

Checkout runs six steps in a row: p95 2.8 s, ~50 bookings a day fail on channel-manager timeouts after the charge. Plan first. Keep hold and charge synchronous. Write the booking and a BookingConfirmed outbox row in one transaction; a relay publishes unsent rows. Invoice, email and loyalty record each eventId in processed_events and skip duplicates. Channel sync retries up to 6 hours, then alerts. Add a test killing the process between commit and publish.

Check before you accept

The booking insert and the outbox insert commit together; no publish() after COMMIT.
Each consumer's test sends the same event twice: one email, one set of points.
On staging, checkout p95 is about 0.5 s and the oldest-unsent-row metric is visible.

Red flags

It commits, then calls the broker: the bug that lost 9 bookings.
It claims the broker guarantees exactly-once, so duplicates don't matter.
It moves the card charge into an async consumer, so "confirmed" shows before payment is known.

Check yourself

2 min

Q1/3Reasoning

Classify each message:

CapturePayment

PaymentCaptured

ReserveRoom

GuestCheckedOut

How confident are you?

Next episode: StayHub wants to switch payment providers, and you find the old provider's SDK imported in 23 files, including the booking rules.

Finished?

Mark it complete to update your skill map.

Next: Layered, hexagonal and clean architecture

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Publishing that survives a crash

You lead the redesign this time; Lina reviews each step. Commands and events look different on purpose:

json

// COMMAND: imperative, one handler, may be rejected
{ "type": "ReserveRoom", "commandId": "c-81f2", "hotelId": 42, "nights": ["2026-07-12", "2026-07-13"] }

// EVENT: past tense, a fact, any number of subscribers
{
  "type": "BookingConfirmed",
  "version": 2,
  "eventId": "e-5c09",
  "occurredAt": "2026-06-01T10:15:03Z",
  "correlationId": "req-77aa",
  "bookingId": "bk_991", "guestId": 81723, "hotelId": 42,
  "totalCents": 31800, "currency": "EUR"
}

The event carries what consumers need, so they don't have to call Booking back.

Worked example

Redesigning StayHub's checkout

Checkout runs six steps in a row: p95 2.8 s, and ~50 bookings a day fail on channel-manager timeouts after the card was charged. Redesign it so the guest waits only for what matters, and nothing gets lost.

1Split the decision from the reactions
Hold room (30 ms) and charge card (400 ms) decide whether a booking exists: keep them synchronous. Invoice, email, loyalty and channel sync are reactions.
Why: The guest needs a yes or no; she doesn't need to wait for the invoice. This alone takes p95 from 2.8 s to about 0.5 s.
2Record the fact with the outbox
Write the confirmed booking and the BookingConfirmed outbox row in one transaction. A relay publishes unsent rows every few hundred milliseconds.
Why: Committing then publishing loses events when the process dies in between. One transaction means "saved" and "announced" can't disagree.
3Make every consumer idempotent
Each consumer records eventId in a processed_events table in the same transaction as its work, and skips duplicates. Loyalty can also upsert by bookingId.
Why: The relay delivers at least once. Without this, a crash means two emails or double points.
4Give the risky step an owner
Channel sync is flaky and matters to hotels (overbooking). Instead of a plain listener, a small orchestrator tracks each booking: pending → pushed or failed, retries with backoff for up to 6 hours, then alerts Hotel Partners.
Why: Pure choreography hides where a step is stuck. A process with retries and a deadline needs state and an owner.
5Tell the guest the truth
Show "Booking confirmed. Your invoice and email will arrive in a minute." The booking page reads from Booking itself, not from a copy updated by events.
Why: Eventual consistency is a product decision. Honest screens and read-your-own-writes prevent guests from booking twice.
6Measure the lag
Dashboard: unsent outbox rows, age of the oldest one, and each consumer's backlog. Alert if the oldest unsent row is older than 60 s.
Why: Async work fails quietly. If you don't measure the delay, you learn about it from hotels.

	Choreography	Orchestration
Who decides the next step?	Each part, reacting to events	A central orchestrator
Coupling	Parts know events, not each other	Orchestrator knows every participant's commands
Seeing where a process is	Spread out; needs tracing	One place holds state and history
Good fit	Simple fan-out: notify, index, analyse	Multi-step processes with retries and compensations
Risk	Hidden chains nobody understands	Orchestrator grows into a "god service"

Interactive simulation

Visual diagram

Guest

Booking

Bookings DB + outbox

Topic: booking-events

Loyalty

Invoicing

Partner sync (new)

Click a component

A booking is confirmed once; many consumers react independently through a topic. The outbox makes publishing reliable.

Analogy

In a hotelIn event-driven design

In a hotelManager: "Clean room 214 now"In event-driven designA command
In a hotelRadio: "Guest left room 214"In event-driven designAn event
In a hotelEach department acts on what it hearsIn event-driven designChoreography
In a hotelA duty manager with a checklistIn event-driven designOrchestration
In a hotelRoom shows vacant, not cleaned yetIn event-driven designEventual consistency

Where the analogy breaks: a radio message is heard once, live. A broker can store events and replay them, and may deliver the same one twice, so listeners must handle repeats.

Check your understanding

Q1/4Reasoning

Checkout calls five dependencies synchronously, each 99.9% available and independent. Roughly what availability does checkout get from them combined? (in %)

How confident are you?

Practice ladder

Step 1 · Trace

Trace a relay crash

Scenario Warm-up

StayHub now uses an outbox. The relay loop is:

Read up to 100 unsent outbox rows
Publish each one to the queue
Mark them all as sent

At 21:03 the relay publishes rows 1–40, then the server is killed before step 3. It restarts at 21:04.

Consumers: Email (no duplicate check), Loyalty (records eventId in processed_events).

Trace what happens to bookings in rows 1–40 after the restart: what is published, what each consumer does, and what guests notice.

Step 2 · Complete

Finish the cancellation redesign

Design Practice

Cancelling a booking today does, synchronously: release the room (20 ms), refund the card (600 ms, sometimes fails and must be retried for up to 3 days), remove loyalty points (60 ms), email the guest (300 ms), notify the hotel's channel manager (500 ms, flaky).

Lina's redesign has two blanks:

Decision (synchronous): ___
Record the fact: BookingCancelled in the outbox, same transaction.
Choreography (simple reactions): loyalty points, guest email.
Orchestrated (needs an owner): ___
Consumers idempotent by eventId.

Fill in steps 1 and 4, and say why each belongs there.

Step 3 · Solve

Events for hotel price changes

Design Practice

Hotels change room prices in the partner extranet about 40,000 times a day. Today, saving a price synchronously: updates the price table, refreshes the search cache, recomputes "deal" badges, and calls an analytics API. Saving takes 1.4 s and fails when analytics is down, so hotel staff retry and sometimes give up.

Redesign it with events. Name the event and its fields, say what stays synchronous, list the consumers, and explain how you handle duplicates and two quick changes to the same room arriving out of order.

Step 4 · Decide

Should the guest confirmation page wait for the invoice?

Decision Stretch

Maya: "Business travellers want the invoice PDF on the confirmation page, not by email a minute later. Can checkout wait for it again?"

Facts: invoice creation takes ~250 ms and fails 0.3% of the time; about 35% of StayHub bookings are business trips; checkout p95 is now 0.5 s.

Write a decision: "We chose X over Y because…, accepting the risk of…". Consider at least one option besides "wait" and "don't wait".

Explain it yourself

Explain to a product manager why the confirmation email might arrive a few seconds after the booking page says 'confirmed', why that's a deliberate design, and what guarantees you still provide.

0 words