Skip to content
Episode 46 · Half a holiday

Sagas and distributed transactions

Thursday, 11:00: two days after StayHub launched hotel + airport transfer packages, 86 guests have been charged about €410 for a package whose transfer was never booked, and their rooms sit in limbo.

A booking touches rooms, payments, points and email, each with its own database. There's no single "save all" button, so here's what you do instead.

30 min advancedSagasTwo-phase commit (2PC)Compensating actionsSaga orchestration vs choreographyTransactional outboxTransactions & ACID
  1. Problem

    Booking a stay means holding a room, charging a card, adding loyalty points and emailing a confirmation, across four services and four databases.

  2. Why the simple system fails

    There is no transaction spanning services. If payment fails after the room is held, or the service crashes halfway, you get ghost holds, charges without bookings, or both.

  3. Engineering concept

    Split the business transaction into local steps, each with a compensating action, and drive them to completion or full compensation (a saga).

  4. Technology

    Local ACID transactions, a saga orchestrator or event choreography, the transactional outbox, idempotent participants; 2PC/XA where a coordinator over a few databases is acceptable.

  5. Trade-offs

    Sagas give up isolation: intermediate states are visible, compensations can fail, and the flow must be designed and observed explicitly.

01

The problem

2 min
Thursday, 11:00. Two days after the "Stay + Transfer" launch.
  1. Maya · Product manager
    86 package guests charged about €410, no airport transfer, room stuck on "pending".
  2. Omar · On-call SRE
    Hold room → charge → book RideCo transfer → confirm. RideCo says "no drivers" 8% of the time. Our code throws there.
  3. You · Engineer at StayHub
    Can't we wrap it all in one transaction?
02

The big idea

4 min
No undo button
The big idea

No shared commit across services: every step needs a planned undo.

Like booking a flight, hotel and car on three websites: if the car is sold out, you cancel the hotel, then the flight. A does that in software, so going back is planned instead of accidental.

03

How it works

5 min

Why not (a coordinator asks everyone "ready?", then says "go")? Participants hold locks while waiting, get stuck if the coordinator dies, and RideCo and the payment provider don't support it anyway.

The package as a saga
  1. 1
    Each step saves for real, with an undo

    A saga is a multi-step action across services. Each step's compensation is its planned undo: release the hold, cancel the transfer, refund. A new action, not an erase.

  2. 2
    Hold the room

    15-minute hold in Inventory. Undo: release it. Even if the undo is lost, the hold expires.

  3. 3
    Book the transfer, early

    Fails 8% of the time and is free to cancel, so it goes before we take money. Most failures then need only the cheap undo.

  4. 4
    Charge: the pivot

    The pivot is the point of no return. Before it, failures compensate in reverse order. After it, steps are retried until they succeed.

  5. 5
    Confirm, then email last

    An email can't be unsent, so it only goes when everything else is done.

  6. 6
    The orchestrator remembers

    The drives the steps and stores each package's progress, so a crash halfway resumes or compensates instead of leaving a mess.

04

Watch a decision being made

4 min
Worked example

You design the package saga; Lina asks the questions

Every package must end fully booked or fully undone, even if RideCo refuses, the provider is down, or our service crashes halfway.

  1. 1List each step and its undo, order by reversibility

    Hold ↺ release → transfer ↺ cancel → charge ↺ refund (pivot) → confirm → email (no undo).

    Why: Lina asks "what's the undo?" A step without one goes after the pivot, where we only push forward.

  2. 2Store progress with the next command

    Each state change and the next command go in one transaction, via the .

    sql
    BEGIN;
UPDATE sagas SET step = 'TRANSFER_BOOKED' WHERE id = 'pk_311' AND step = 'HELD';
INSERT INTO outbox (topic, key, payload)
VALUES ('payments.commands', 'pk_311', '{"type":"Charge","key":"pk_311-charge"}');
COMMIT;

    Why: Update-then-publish is a dual write: a crash between them loses the command or sends one for a state that never committed.

  3. 3Make every step and undo retry-safe

    Keys like pk_311-charge, pk_311-refund. Timeouts and 503s retry with backoff; only a permanent refusal starts compensation.

    Why: The orchestrator will repeat steps after crashes. Unknown outcomes are retried with the same key, never guessed.

  4. 4Name the halfway states

    Packages show as PENDING until confirmed. The hold's 15-minute TTL is the backstop.

    Why: A saga has no isolation: others can see halfway states, so another flow must never treat a held room as booked.

The whole thing in one line: Four calls became a saga: undos planned, risky steps before the charge, progress saved with each command, every step repeatable. Lina adds one alert: sagas stuck over 10 minutes.
05

In real life

3 min
Hide the steps versus manage the steps

Google Spanner: transactions across machines

  • Publicly described as running ACID transactions across many servers, with two-phase commit inside the database
  • The developer writes one transaction
  • Only covers data stored in that database

Fits when every record lives in one system you control.

Uber Cadence: workflows with compensation

  • Publicly open-sourced (Temporal grew from it): persists each step and retries it
  • Each service keeps its own data; the workflow drives steps and undos
  • Reaches across services and outside companies

Fits flows that cross services or companies.

06

What goes wrong

3 min
Compensations that run once

The refund API can be down too; the saga ends half-undone and marked done.

Instead

Retry undos with backoff and keys; stay COMPENSATING until all succeed; alert after 10 minutes.

Irreversible steps early

Emailing "confirmed" before charging means a "sorry, cancelled" email when payment fails.

Instead

Cheap, likely-to-fail steps first; irreversible ones last.

Ignoring the missing isolation

Another flow reads a held package as booked.

Instead

Explicit PENDING states, holds with TTLs.

07

Trade-offs

3 min
ApproachFits whenYou pay
One database transactionOne team, one domainCouples domains; can't reach outside APIs
2PC / XAA few databases you controlStuck participants; few APIs support it
Orchestrated sagaCritical flows: checkout, packagesA central component to run
Choreographed saga (events)Side effects that can lagFlow implicit, hard to trace
Monday, 10:00. Choosing how to drive the saga.
  1. Karim · CTO
    We already run Kafka. Each service reacts to the previous event. No new orchestrator to run.
  2. Lina · Senior engineer, your mentor
    Then the flow lives in five subscriptions across three teams. When a package is stuck, who knows which event never came?
  3. Karim · CTO
    A dashboard on the events. And no single component that can take every package down.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • The order, by reversibility: hold (15-minute TTL) → transfer → charge (pivot) → confirm → email last.
  • Which errors trigger compensation (a permanent refusal) and which only retry (timeouts, 503).
  • What guests and other flows see in between: a named PENDING state, never "booked" early.
Tell Claude Code
86 StayHub packages ended charged with no transfer because the code threw when RideCo refused. Plan an orchestrated saga first. Steps: hold room (15-minute TTL), book transfer, charge (pivot), confirm, email last. Steps before the pivot have compensations, run in reverse. Every step and undo uses a key like <packageId>-refund. Timeouts and 503s retry with backoff; compensations retry until they succeed. Save saga state and the next command in one transaction via the outbox. Show PENDING until confirmed. Test RideCo refusing and a refund 503.
Check before you accept
  • In the step list, the transfer comes before the charge and the email comes last.
  • A compensation is retried with the same key until it succeeds, not called once inside a try.
  • The next command goes into the outbox in the same transaction as the state change.
Red flags
  • It calls the refund once and logs the error if it fails.
  • It proposes 2PC with RideCo or the payment provider.
  • It treats a RideCo timeout as a refusal and compensates while the transfer may be booked.
09

Check yourself

2 min
Q1/3Scenario
Steps 1–3 of a saga succeeded: hold the room, reserve a parking spot, charge the card. Step 4 (confirm the booking with the hotel's system) failed permanently. Put the compensations in order:
  1. 1.Refund the charge
  2. 2.Release the parking reservation
  3. 3.Release the room hold
How confident are you?

Next episode: two copies of the nightly payout job both think they're in charge, and some hotels get paid twice.

Finished?

Mark it complete to update your skill map.

Next: Consensus, leader election and distributed locks
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

An orchestrator that can't forget
Worked example

You design the package saga; Lina asks the questions

Redesign "Stay + Transfer" so every package ends either fully booked or fully undone, even if RideCo refuses, the provider is down, or our service crashes halfway.

  1. 1List each step and its undo

    Hold room ↺ release hold. Book transfer ↺ cancel transfer. Charge ↺ refund. Confirm ↺ cancel booking. Email: no undo.

    Why: Lina's question: "what's the undo?" A step without one must go after the pivot, where we only push forward.

  2. 2Order by reversibility

    Hold → transfer → charge (pivot) → confirm → email.

    Why: Likely failures and cheap undos first, the expensive-to-undo charge in the middle, the irreversible email last.

  3. 3Store progress with the next command

    The orchestrator keeps a sagas row per package. Each state change and the next command go in one transaction, via the .

    Why: Update-then-publish is a dual write: a crash between them leaves a saga stuck or sends a command for a state that never committed.

  4. 4Make every step and undo retry-safe

    Keys like pk_311-charge and pk_311-refund. Transient errors (timeouts, 503) retry with backoff; only a permanent refusal triggers compensation.

    Why: The orchestrator will repeat steps after crashes. Episodes two and four: unknown outcomes are retried with the same key, never guessed.

  5. 5Name the halfway states

    Packages show as PENDING to the guest and to other flows until confirmed. The hold has a 15-minute TTL.

    Why: A saga has no isolation: others can see halfway states. Naming them stops another flow from treating a held room as booked, and the TTL cleans up if everything else fails.

The whole thing in one line: You turned four calls into a saga: each step has an undo, failure-prone steps come before the charge, progress and next command are saved together, and every step can be safely repeated. Lina only added an alert on sagas stuck for more than 10 minutes.
ts
const packageSaga: Step[] = [
  { name: "hold",     run: (c) => inventory.hold(c.roomId, c.dates, { key: `${c.id}-hold`, ttlMin: 15 }),
                      compensate: (c) => inventory.release(c.roomId, { key: `${c.id}-release` }) },
  { name: "transfer", run: (c) => rideco.book(c.flight, c.hotel, { externalRef: `${c.id}-transfer` }),
                      compensate: (c) => rideco.cancel({ externalRef: `${c.id}-transfer` }) },
  { name: "charge",   run: (c) => payments.charge(c.amount, { key: `${c.id}-charge` }),
                      compensate: (c) => payments.refund(c.amount, { key: `${c.id}-refund` }) },
  // pivot passed: everything below must eventually succeed (retry, never compensate)
  { name: "confirm",  run: (c) => bookings.confirm(c.id) },
  { name: "email",    run: (c) => email.sendConfirmation(c.id, { key: `${c.id}-email` }) },
];
Each step carries its own key. Steps after the pivot have no compensation: they're retried until they succeed.
Local ACID transaction2PC / XASaga
All-or-nothingYesYesEventually: completes or compensates
IsolationYesYes (locks held while waiting)No: halfway states are visible
Works across services and partnersNoOnly participants that support itYes, any API with an undo
When the coordinator failsn/aParticipants can be stuck holding locksOrchestrator resumes from stored state
Where the complexity livesThe databaseThe transaction managerYour code: undos, state, monitoring
A saga keeps atomicity (eventually) and durability, and gives up isolation. You design for that on purpose.
11

Interactive simulation

12

Visual diagram

Click a component

An orchestrated booking saga. Payment succeeds, loyalty fails permanently: watch the compensations.

13

Analogy

  • Booking a trip yourselfFlight, hotel and car on three websitesA sagaThree services, three databases
  • Booking a trip yourselfNo "book all or none" buttonA sagaNo shared transaction
  • Booking a trip yourselfCar sold out after booking the restA sagaA later step fails
  • Booking a trip yourselfCancelling the hotel, then the flightA sagaRunning compensations in reverse
  • Booking a trip yourselfYou with a checklist, driving it allA sagaAn orchestrator

Where the analogy breaks: you notice a sold-out car and react. A saga must handle crashes mid-way, so it records progress and can resume or undo after a restart.

14

Check your understanding

Q1/3Reasoning
In two-phase commit, a participant has voted 'prepared' and then loses contact with the coordinator. What can it safely do?
How confident are you?
15

Practice ladder

Step 1 · Trace

Trace a package that fails at confirm

Scenario Warm-up

Package pk_412 runs the saga (hold → transfer → charge → confirm → email). This version still treats a permanent confirm failure as a reason to compensate.

  • 10:00:00 hold succeeds.
  • 10:00:02 transfer succeeds.
  • 10:00:04 charge times out.
  • 10:00:05 charge retried with key pk_412-charge: provider returns the original success.
  • 10:00:06 confirm fails permanently: the hotel's room type was deleted.

List the compensations in the order they run, with the key each uses, and the saga's final state. What does the guest receive?

Step 2 · Complete

Order the hotel + dinner saga

Design Practice

StayHub adds a "Stay + Dinner" package. The steps, shuffled:

  • A. Send the confirmation email
  • B. Reserve a dinner table at the hotel's restaurant (free to cancel up to 2 h before; fails about 5% of the time when full)
  • C. Charge the card
  • D. Hold the room (15-minute TTL)
  • E. Confirm the booking

Put them in order, mark the pivot, and give the compensation for each step before the pivot.

Step 3 · Solve

Change the dates of a paid booking

Design Practice

Guests can now move a paid booking to new dates. A change from 12–14 May (€300) to 19–22 May (€420) needs:

  • Hold the new nights.
  • Charge the €120 difference.
  • Release the old nights.
  • Update the booking and email the guest.

Old nights released too early could be sold to someone else; if the new charge then fails, the guest has lost their original stay. Design the saga: order, compensations, pivot, and what the guest sees while it runs.

Step 4 · Decide

Saga or one database for paying with points?

Decision Stretch

Guests will be able to pay part of a booking with loyalty points. Points live in the Loyalty service's own database (Loyalty team); bookings in the Booking service's database (Booking team).

  • Option A, saga: reserve points in Loyalty → charge the rest by card → confirm; compensations: release points, refund.
  • Option B, move the points ledger into the Booking database, so reserving points and confirming the booking is one local transaction.

Facts: the Loyalty team also runs partner promotions and point expiry on that ledger. About 4% of bookings will use points.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain why a booking that spans Inventory, Payment and Booking services can't just use a database transaction, and how a saga keeps it consistent.

0 words