CAP, PACELC and consistency
Wednesday, 9:20: a guest books a room, lands on "My bookings", and the list is empty. The booking is safely in the database, yet since Monday 63 guests have booked twice because of this.
When two copies of your data can't talk, you must pick: refuse to answer, or answer and risk being wrong.
- Problem
You keep copies of data on several machines for speed and survival. The copies must agree, but agreeing requires talking over a network that can fail.
- Why the simple system fails
Pretending copies are always in sync gives users stale or conflicting data; insisting they always are makes the system stop whenever the network hiccups, and slow even when it doesn't.
- Engineering concept
CAP: during a partition, choose consistency or availability. PACELC: otherwise, choose latency or consistency. Pick guarantees per operation, and give users at least read-your-writes.
- Technology
Leader/follower replication, synchronous vs asynchronous replication, quorums, session stickiness, version numbers and conflict resolution.
- Trade-offs
Correctness vs uptime during failures; freshness vs latency and cost every day.
The problem
2 min- Maya · Product managerSofia booked in Porto, opened My bookings, saw nothing, and booked again. 63 doubles since Monday.
- Omar · On-call SREBoth of her bookings are in the primary database. Nothing failed.
- Lina · Senior engineer, your mentorGuess first. Hint: My bookings doesn't read from the primary.
The big idea
4 minCopies trade freshness for speed every day, and correctness for uptime when the network splits. Choose the guarantee per piece of data, by the cost of a stale read.
Two front desks keep their own room boards in sync by phone. On a busy morning, desk B doesn't show a guest who just checked in at A. If the line dies, each desk must refuse check-ins or risk giving the same room twice.
How it works
5 minWhy not read everything from the primary? At peak, ~3,000 availability lookups a second. Replicas exist so the primary only does the work that must be exact. Replication lag is how far a copy trails the primary.
- 1Write on the primary
The booking commits. Sofia gets "Confirmed".
- 2Ship the change
The primary streams its change log to three replicas without waiting ().
- 3Replicas apply it
Usually 50 ms later; during the 9:00 price imports, up to 2.5 s.
- 4Read from a replica
My bookings reloads 0.5 s later from a replica 2 s in the past.
- 5Guest acts on it
Sofia sees nothing and books again. Both are real.
names the everyday half: else (no partition), choose latency or consistency. The is the bad day: in a , machines run but can't reach each other, and each copy must refuse (consistency) or answer and maybe be wrong (availability).
Sofia's fix is narrow: , you always see your own changes, even if others' view lags.
Watch a decision being made
4 minLina chooses a guarantee per piece of data
Replicas lag up to 2.5 s at 9:00 and guests book twice. Decide what each kind of data needs, and fix My bookings without moving all reads to the primary.
- 1Measure the lag
A week's graph: about 50 ms most of the day, 2.5 s during the 9:00 imports. All the doubles fall in that window.
Why: "A bit behind" isn't a number. Size and timing tell you which screens can notice.
- 2Price a stale read for each reader
Rooms left at booking: sold twice. My bookings and profile: the owner thinks it's lost. Reviews, descriptions, search: nobody notices.
Why: The cost of being wrong, not the data's importance in general, picks the guarantee.
- 3Pick a guarantee
Rooms left: strong, decided on the primary. My bookings and profile: read-your-writes. The rest: eventual, from replicas and caches.
Why: Strong costs load and latency, so spend it only where a stale read loses money or trust.
- 4Build read-your-writes
After a write, keep the primary's log position in the session for 10 s. In that window, read a replica only if it has replayed past it; otherwise the primary.
textwrite → session.lsn = primary_lsn(), expires in 10 s read → replica.replayed_lsn >= session.lsn ? replica : primaryWhy: Only the guest who just wrote pays the check, only for seconds. Everyone else still reads replicas.
In real life
3 minAmazon Dynamo: always accept the write
- The 2007 paper publicly describes a cart writable even during failures
- Conflicting carts are merged later; a removed item can come back
Fits when refusing costs more than an occasional merge.
Google Spanner: always agree
- Publicly described as globally distributed with strongly consistent transactions
- Writes wait for a majority; without one, they wait or fail
Fits when a wrong answer costs more than a slow one.
What goes wrong
3 minAcross machines, partitions aren't optional; ignoring them means undefined behaviour.
Assume partitions; decide what each operation does in one.
Strong everywhere is slow; eventual everywhere double-sells.
Choose per operation, by the cost of a stale read.
Machine clocks disagree; a later edit silently loses.
Version numbers or conditional writes.
Trade-offs
3 min| Guarantee | Fits when | You pay |
|---|---|---|
| Strong (CP / PC) | Being wrong costs money: inventory, payments, balances | Slower writes; the minority side refuses |
| Eventual (AP / EL) | Stale data is cheap: content, search, counters | Stale reads; conflict rules |
| Read-your-writes | Users edit their own data, most reads hit replicas | Session routing; others may still lag |
- Karim · CTOIf one zone is cut off, that side refuses bookings: revenue walking away. Let every side accept and fix overbookings by hand.
- Lina · Senior engineer, your mentorAn overbooking is a guest turned away and a relocation we pay for. I'd rather show "try again in a minute".
- Karim · CTOHotels overbook on purpose all the time.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The guarantee per kind of data: rooms left strong, My bookings and profile read-your-writes, the rest eventual. An agent optimising for speed routes everything to replicas.
- How much staleness you accept, from measured lag (~50 ms, 2.5 s at 9:00), and the read-your-writes window (10 s, then the primary).
- How conflicting edits are settled: versions or conditional writes, never last-write-wins by clock.
Since Monday, 63 guests booked twice because My bookings reads an async replica lagging up to 2.5 s at 9:00. Plan first. Don't move all reads to the primary, and keep the availability check on the primary. After a booking or profile write, store the primary's log position in the session for 10 s; in that window read the guest's own data from a replica only if it has replayed past it, otherwise the primary. Show which queries change, and test 3 s of simulated lag.
- Claude lists every changed query and its copy; availability still reads the primary.
- With no caught-up replica in the window, reads fall back to the primary.
- The lag test shows the new booking immediately, and primary load rises only in the 10 s windows.
- The availability check moved to a replica "for speed".
- Every read moved to the primary, or replication made synchronous.
- A cache in front of the replicas to "fix" staleness.
Check yourself
2 minNext episode: a guest's phone loses signal mid-payment, the app retries, and the bank statement shows the same €360 twice.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Choosing a guarantee per piece of data
Replicas lag up to 2.5 s at 9:00, and guests are booking twice. Decide what each kind of StayHub data needs, then fix My bookings without moving all reads to the primary.
- 1Measure the lag
Graph replica lag over a week: about 50 ms most of the day, 2.5 s during the 9:00 price imports. The doubles all fall in that window.
Why: "Replicas are a bit behind" isn't a number. The size and timing of the lag tell you which screens can notice it.
- 2List who reads what
Rooms left at the booking step; My bookings; guest profile; hotel descriptions; reviews; search results.
Why: Guarantees are chosen per operation. A single system-wide setting is either too slow or too loose.
- 3Price being stale
Ask, for each: what if a guest reads a copy 2.5 s old? Rooms left: sold twice. My bookings and profile: the owner thinks it's lost. Reviews, descriptions, search: nobody notices.
Why: The cost of being wrong, not the data's importance in general, picks the guarantee.
- 4Pick a guarantee
Rooms left at booking: strong, decided on the primary. My bookings and profile: read-your-writes. Reviews, descriptions, search: eventual, from replicas and caches.
Why: Strong costs load and latency, so it's spent only where a stale read loses money or trust.
- 5Build read-your-writes
After a write, store the primary's log position in the session for 10 s. Reads in that window use a replica only if it has replayed past that position; otherwise the primary.
Why: Only the guest who just wrote pays the extra check, and only for a few seconds. Everyone else still reads replicas.
Quorums make the dial explicit. With N copies, a write waits for W acknowledgements and a read asks R copies. If R + W > N, every read overlaps the latest write. StayHub's inventory cluster uses N = 3, W = 2 across three zones.
| N, W, R | R + W > N? | Behaviour | During a partition |
|---|---|---|---|
| 3, 3, 1 | Yes | Slow writes (wait for all), fast reads | Any one copy unreachable → writes fail |
| 3, 2, 2 | Yes | Balanced; survives one slow or dead copy | Majority side keeps working; minority refuses |
| 3, 1, 1 | No | Fastest; reads can be stale | Both sides accept writes → conflicts to resolve later |
UPDATE bookings
SET check_out = '2026-11-16', version = version + 1
WHERE id = 8812 AND version = 7;
-- 1 row -> our write won
-- 0 rows -> someone else changed it first: reload, show the user, retryInteractive simulation
Visual diagram
Two replicas, a broken link between them, and the choice each replica must make when a write arrives.
Analogy
- Two front desksEach desk has a copy of the room boardTwo database copiesReplicas in two places
- Two front desksThe phone line between them diesTwo database copiesA network partition
- Two front desks"Sorry, I can't confirm right now"Two database copiesChoosing consistency
- Two front desks"Sure!" and hoping for the bestTwo database copiesChoosing availability
- Two front desksCalling the other desk before every bookingTwo database copiesPACELC: trading speed for correctness
Where the analogy breaks: two desks can chat and fix mistakes by hand. Database copies need clear rules decided in advance, like version numbers, to settle disagreements automatically.
Check your understanding
Practice ladder
Trace three reads after a booking
Scenario Warm-upStayHub's read-your-writes code is live. Sofia books at 9:10:00; the primary's log position after her write is 0/16B3A00, saved in her session for 10 s.
At 9:10:01 three reads happen:
| Who | Reads | Replica picked | Replica has replayed up to |
|---|---|---|---|
| Sofia | My bookings | replica-1 | 0/16B39F0 |
| Sofia | My bookings (refresh) | replica-2 | 0/16B3A10 |
| Tom (no recent writes) | Hotel 42's reviews | replica-1 | 0/16B39F0 |
For each read, say which machine serves it and whether the result includes Sofia's booking (where relevant). Then: what happens to Sofia's reads at 9:10:15?
Finish Lina's guarantee table
Design PracticeLina's table assigns a guarantee to each kind of StayHub data. Two rows are blank. Fill in the guarantee and a one-line reason for each.
| Data | Guarantee | Why |
|---|---|---|
| Rooms left, at the booking step | Strong (primary, conditional update) | A stale read sells a room twice |
| Rooms left, on search results | Eventual (replica / cache) | It's a hint; the booking step re-checks |
| Guest's email address, on the profile page | ___ | ___ |
| Payment status of a booking (paid / refunded) | ___ | ___ |
| Number of reviews on a hotel card | Eventual | Off by one for a few seconds hurts nobody |
The hotel owner's price didn't change
Debugging PracticeNadia runs a hotel in Lisbon and edits prices in StayHub's hotel extranet. She raises Saturday from €140 to €180, clicks Save, and the preview page shows €140. She saves again; it now shows €180. She calls Maya, annoyed.
Facts: - Price writes go to the primary. The preview page reads a replica. - Search results come from a search index updated from price events, with a delay of about 30 s. - Guests who booked Saturday in those 30 s paid €140.
Explain both problems. Propose the right guarantee for the preview page, for search, and for the price charged at booking.
Exact "rooms left" on every search?
Decision StretchMaya: "Guests click a hotel showing '2 rooms left' and get 'sold out' at the booking step. It happens about 40 times a day. Can search show the exact number, straight from the primary?"
Facts: about 50 searches a second at peak, each showing 20 hotels for a ~3-night stay (one availability row per hotel and night). The primary handles bookings and price imports and runs at 55% CPU at peak. The booking step already re-checks on the primary.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain CAP and PACELC using a hotel with two replicas, and say which guarantee you'd give inventory, reviews and a guest's profile.