Episode 41 · Loyalty points broke checkout

Kafka and event-driven systems

Monday, 10:40: the loyalty points service is having a bad morning, and checkout now takes 6 s with 12% of bookings failing. Loyalty points, not payments, and Karim wants to know how that's even possible.

Instead of phoning five services after every booking, write it down once and let each one read it when it's ready.

30 min advancedEvent-driven architectureQueue vs event logKafka topics & partitionsConsumer groupsAt-least-once deliveryIdempotency

Problem
When a guest books, five other systems must react: email, analytics, loyalty, notifications and fraud checks.
Why the simple system fails
If Booking calls each one directly, every new consumer means changing Booking, and any slow or failing consumer slows or fails the booking itself.
Engineering concept
Booking records a fact, BookingCreated, in a durable log. Each interested system reads it independently, at its own pace, and can replay history.
Technology
Apache Kafka: topics, partitions, keys, consumer groups, offsets, retention; idempotent consumers; the transactional outbox.
Trade-offs
Decoupling and replay in exchange for eventual consistency, duplicate deliveries, harder debugging, and a serious system to operate.

The problem

2 min

Monday, 10:40. The checkout dashboard on the big screen.

Omar · On-call SRE
Checkout p99 is 6 s. 12% of bookings fail. All errors: Loyalty timed out.
Karim · CTO
Loyalty points? That's a nice-to-have. How can it fail a booking?
Lina · Senior engineer, your mentor
Make Loyalty faster today, and next week it's Fraud or Email. Guess what's really wrong.

The big idea

4 min

How points broke checkout

The big idea

Write the fact once to a durable log, answer the guest, and let every service read it at its own pace, including after a bad morning.

Today the front desk phones five departments after every check-in and stays on the line until each answers. With an log, it pins one note on a shared board and serves the next guest; each department reads the board when it can.

How it works

5 min

In , a topic is one log of events, like bookings, split into so readers work in parallel. Each service reads with its own consumer group, which keeps its own bookmark: the .

One booking, with an event log

1
Save the booking
Booking writes the booking and an row in one transaction: about 80 ms.
2
Answer the guest
"Confirmed" right away. Nobody else has been called.
3
Publish the fact
A relay appends BookingCreated to the bookings topic, keyed by booking id.
4
Each group reads
Email, Analytics, Loyalty, Notifications and Fraud each read every event.
5
Slow readers catch up
If Loyalty is down an hour, its offset stays put; when it's back, it reads what it missed.

Checkout drops to about 90 ms instead of 540 ms on a good day. The price: points arrive seconds later, and delivery is : never lost once written, but maybe received twice, so every reader must cope with duplicates.

Watch a decision being made

4 min

Worked example

Lina moves Loyalty off the checkout path

Move the five calls to Kafka without losing a booking event and without giving anyone double points.

1Publish a fact, written with the booking
One event, BookingCreated v2 with an eventId, inserted into the outbox in the booking's transaction; a relay publishes it.
sql
```
BEGIN;
INSERT INTO bookings (...) VALUES (...);
INSERT INTO outbox (event_id, key, payload) VALUES ('evt_01JB…', 'bk_8812', '{...}');
COMMIT;
```
Why: A fact can have any number of readers; a command ("AwardPoints") ties Booking to Loyalty again. Database-then-Kafka is two writes that a crash can split.
2Pick the key and partitions
Topic bookings, 12 partitions, key = bookingId: Created, Paid and Cancelled for one booking stay in order.
Why: Kafka keeps order only inside a partition, and 12 partitions allow up to 12 parallel readers per service.
3A group per service, every reader idempotent
Groups email, loyalty, fraud…; retention 7 days. Each reader records eventId in processed_events in the same transaction as its effect, then commits the offset.
Why: Groups don't disturb each other, and 7 days lets Fraud replay last week. Crashes and rebalances redeliver events.
4Watch the lag
Alert when a group's stays above 5 minutes; failing events go to a after 5 tries.
Why: Async problems are silent: guests see nothing while points pile up. Lag is the new error rate.

In real life

3 min

Two well-known companies, two choices

LinkedIn: an event log for everything

Created Kafka and has publicly described it as its data backbone
Many teams read the same streams and replay history; dedicated infrastructure teams run it

Fits many independent teams needing the same facts.

37signals: jobs in the database

Open-sourced Solid Queue, a job queue in the database they already run
Each job has one known handler; no cluster to operate

Fits background jobs with one handler each.

What goes wrong

3 min

Random or missing keys

One booking's events spread across partitions: Cancelled before Created.

Instead

Key by what must stay ordered (bookingId).

Using events for request and answer

"Publish CheckAvailability, wait for the reply" is a fragile phone call in disguise.

Instead

Call directly when you need an answer now.

Kafka for one background job

Partitions, retention and lag monitoring just to send emails.

Instead

A job queue or outbox + worker until several readers need the events.

Trade-offs

3 min

Style	Fits when	You pay
Synchronous calls	You need the answer to continue (price, payment)	Caller waits for the slowest; failures cascade
Job queue / outbox + worker	One or a few known handlers, modest volume	Weak fan-out, no replay
Event log (Kafka)	Several teams need the same facts, or replay	A serious system to run; duplicates and lag

Friday, 11:00. Karim has the managed Kafka quote.

Karim · CTO
About €1,800 a month, and nobody here has run it. We have a Postgres outbox. Why not one job per service per booking?
Lina · Senior engineer, your mentor
Then Booking must know every reader, and the sixth service asks us again. And jobs are gone once done: no replay for Fraud.
Karim · CTO
I'd rather pay in a little coupling than in a new system to be on call for.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Whether you need Kafka at all, and which steps leave the checkout path (maybe Fraud must still block).
The key and partition count (bookingId, 12): it fixes ordering and maximum parallel readers, and is painful to change.
Retention (7 days), dead-lettering after 5 tries, and the lag you alert on (5 minutes).

Tell Claude Code

Move the five post-booking calls off checkout. Plan first, wait for approval. Booking writes BookingCreated v2 with an eventId to an outbox in the booking's transaction; a relay publishes to topic bookings, key bookingId (12 partitions, 7-day retention). Start with the loyalty consumer only: group loyalty, record eventId in processed_events in the same transaction as the points, commit the offset after. Dead-letter after 5 attempts. Test a duplicate delivery (points once) and add a lag alert above 5 minutes.

Check before you accept

The outbox insert is inside the booking transaction; no kafka.send after the commit.
The producer key is bookingId, and the loyalty handler records eventId and points in one transaction before committing the offset.
Checkout goes from ~540 ms to around 90 ms.

Red flags

Offsets committed before the work, or "Kafka is exactly-once, no dedup needed".
More consumers in a group than partitions, expecting more speed.
Commands like AwardPoints instead of facts like BookingCreated.

Check yourself

2 min

Q1/3Scenario

Booking synchronously calls Email, Analytics, Loyalty, Notifications and Fraud. Loyalty's database is down for 20 minutes and bookings start failing. Which change addresses the root cause?

How confident are you?

Next episode: the payment provider stops answering mid-checkout, and nobody can tell whether the guest was charged.

Finished?

Mark it complete to update your skill map.

Next: Partial failures: when part of the system breaks

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Publishing safely, reading safely

Let's look at what Booking actually writes. An is a past-tense fact with an id and a version, so readers can deduplicate and the shape can evolve:

json

{
  "eventId": "evt_01JB7Q8K3V2",
  "type": "BookingCreated",
  "version": 2,
  "occurredAt": "2026-10-01T14:03:22Z",
  "data": { "bookingId": "bk_8812", "hotelId": 42, "guestId": 77, "totalCents": 36000, "currency": "EUR", "checkIn": "2026-11-12", "nights": 3 }
}

A fact, not a command: it says what happened, not what anyone should do.

Worked example

Lina moves Loyalty off the checkout path

Checkout calls five services in a row and fails when Loyalty is slow. Move the five calls to Kafka without losing a booking event and without giving anyone double points.

1Name the fact
Define one event, BookingCreated v2, with eventId, bookingId, guestId, hotelId and the total. Booking publishes facts; it doesn't tell Loyalty what to do.
Why: A fact can have any number of readers. A command ("AwardPoints") ties Booking to Loyalty again.
2Write it with the booking
In the same database transaction as the booking, insert the event into an outbox table. A small relay process publishes outbox rows to Kafka and marks them sent.
sql
```
BEGIN;
INSERT INTO bookings (...) VALUES (...);
INSERT INTO outbox (event_id, key, payload) VALUES ('evt_01JB…', 'bk_8812', '{...}');
COMMIT;
```
Why: Writing to the database and then to Kafka is two systems. A crash in between loses the event or announces a booking that rolled back. The makes them one write.
3Pick the key and the partitions
Topic bookings, 12 partitions, key = bookingId. Created, Paid and Cancelled for one booking land in one , in order.
Why: Kafka only keeps order inside a partition. Within a consumer group, each partition goes to one reader, so 12 partitions allow up to 12 parallel readers per service.
4Give each service its own group
Consumer groups email, analytics, loyalty, notifications, fraud. Each keeps its own . Retention: 7 days.
Why: Groups don't disturb each other: a slow Analytics doesn't slow Email. Seven days of retention lets Fraud replay last week.
5Make every reader idempotent
Each reader records eventId in a processed_events table in the same transaction as its effect, and skips events it has seen. Offsets are committed after the transaction.
Why: Kafka is : crashes and rebalances redeliver events. The relay can also publish a row twice if it dies before marking it sent.
6Watch the lag
Alert when a group's (events written but not yet read) stays above 5 minutes. Move events that keep failing to a after 5 tries.
Why: Async problems are silent: the guest sees nothing while points pile up unprocessed. Lag is the new error rate.

// Relay: publish outbox rows. Key = bookingId keeps per-booking ordering.
await producer.send({
  topic: "bookings",
  messages: rows.map((r) => ({ key: r.bookingId, value: JSON.stringify(r.event) })),
});
await db.query("UPDATE outbox SET published_at = now() WHERE id = ANY($1)", [rows.map((r) => r.id)]);
// Crash between these two lines → rows are published again → readers must dedupe.

Even the outbox relay can publish twice. That's why the readers deduplicate.

	Job queue (SQS, RabbitMQ, a Postgres table)	Event log (Kafka, Kinesis, Redpanda)
Model	A task for one worker	A fact for any number of readers
After processing	Message deleted	Event kept until retention expires
Many readers	One queue per reader to set up	Built in: one consumer group each
Replay history	No	Yes, move a group's offset back
Ordering	Usually none, or limited FIFO	Per partition (per key)
Running it	Simple; many managed options	More to learn: partitions, retention, rebalancing, schemas

Queues hand out tasks. Logs record facts.

Interactive simulation

Visual diagram

Guest

Booking service

Bookings DB + outbox

Topic: bookings (12 partitions)

Email consumers

Loyalty consumers

Analytics

Fraud scoring

Click a component

Booking publishes once; every consumer group reads independently.

Analogy

At a hotel front deskIn Kafka

At a hotel front deskPhoning every department after check-inIn KafkaDirect calls to every service
At a hotel front deskWriting it in the shared logbookIn KafkaPublishing an event to a topic
At a hotel front deskOne logbook section per floorIn KafkaPartitions, read in parallel
At a hotel front deskEach department with its own bookmarkIn KafkaConsumer groups with their own position
At a hotel front deskA new concierge reading last monthIn KafkaReplaying old events

Where the analogy breaks: a paper logbook never shows the same line twice. Kafka can deliver an event more than once, so every reader must handle repeats safely.

Check your understanding

Q1/4Reasoning

The bookings topic has 12 partitions. To handle a backlog you scale the analytics consumer group to 16 instances. How many instances sit idle?

instances

How confident are you?

Practice ladder

Step 1 · Trace

Trace a crash in the email group

Scenario Warm-up

The bookings topic has 12 partitions. Partition 7 holds these events (offsets on the left):

Offset	Event
1040	BookingCreated bk_8812
1041	BookingPaid bk_8812
1042	BookingCreated bk_9001
1043	BookingCancelled bk_8812

The email group's committed offset for partition 7 is 1041 (meaning 1040 is done, 1041 is next). The email reader sends the emails for 1041 and 1042, then crashes before committing. The loyalty group is at offset 1044.

Trace what happens when the email reader restarts. Which emails might go out twice? Is Loyalty affected?

Step 2 · Complete

Finish Lina's Notifications reader

Code Practice

The Notifications service pushes "New booking" alerts to the hotel's app. Lina started the reader but left two gaps. Fill in gap A (skip events already handled) and gap B (when to commit the offset), and say why the order matters.

await consumer.run({
  autoCommit: false,
  eachMessage: async ({ topic, partition, message }) => {
    const evt = JSON.parse(message.value!.toString());
    await db.transaction(async (tx) => {
      // gap A: ___
      await tx.query(
        "INSERT INTO hotel_alerts (hotel_id, booking_id, kind) VALUES ($1, $2, $3)",
        [evt.data.hotelId, evt.data.bookingId, evt.type]);
    });
    // gap B: ___
  },
});

Step 3 · Solve

Ghost reservations in the hotel app

Debugging Practice

Nadia, who runs a hotel in Lisbon, calls Maya: her app sometimes shows bookings that were cancelled hours ago. Omar finds that the Notifications reader sometimes processes BookingCancelled before BookingCreated for the same booking. The cancel finds nothing to remove, then the create adds the alert, and it stays forever.

A new service, Partner Sync, publishes these events to the bookings topic with no key. The topic has 12 partitions; the notifications group runs 4 readers.

Explain the cause and fix it. Then say what you'd do about the ghost alerts already in Nadia's app.

Step 4 · Decide

Kafka for invoice PDFs?

Decision Stretch

The Finance team wants a monthly PDF invoice sent to each partner hotel: about 4,000 PDFs on the 1st of each month, generated by one Finance service. They ask for a new Kafka topic invoice-requests, "since we have Kafka now".

StayHub also has a Postgres outbox + worker setup that the Email service still uses for password-reset emails.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain why a booking platform might move from direct service calls to Kafka events, and how you'd decide whether Kafka is worth it yet.

0 words