Redundancy and failover
Wednesday, 2:14 a.m.: the database host in zone A dies. A healthy standby sits ready in zone B, but nothing switches to it, and every booking fails for 41 minutes while Omar wakes up and follows a runbook.
Machines break all the time. Here's how to keep the service running when one does, without making things worse.
- Problem
At 2:14 a.m. the database host's disk controller dies. Every booking fails until someone wakes up, diagnoses it and rebuilds the server, about two hours later.
- Why the simple system fails
One copy of anything is a single point of failure. A second copy that nobody can switch to quickly, or that switches by itself at the wrong moment, doesn't fix it.
- Engineering concept
Run spare capacity (N+1) in independent failure domains, detect failures automatically with health checks, and fail over to a standby or shift load to active peers, with a quorum and fencing so only one leader ever accepts writes.
- Technology
Load balancer health checks, multi-AZ instance groups, database standbys with automatic promotion (managed services, Patroni-style managers), witnesses and quorums, fencing, DNS or proxy re-routing.
- Trade-offs
Redundancy costs idle capacity and complexity. Fast failover risks false positives and split brain; cautious failover means longer outages. Async standbys can lose the last writes.
The problem
2 min- Omar · On-call SREPrimary died at 02:14. Bookings failed for 41 minutes. The standby was fine; I was the slow part.
- Karim · CTO41 minutes is almost the whole month's budget. Make it automatic.
- Omar · On-call SREEasy: if the standby can't reach the primary for 10 seconds, it promotes itself.
- Lina · Senior engineer, your mentorYou draft the design this time. What could that rule do wrong?
The big idea
4 minRedundancy needs automatic , and failover needs a majority and fencing. A spare nobody can switch to is just an expensive spare part.
The machine part of Wednesday night took seconds; the human part took 41 minutes. Automate it, but safely.
How it works
5 min- 1Detect
Health checks every 10 s; the primary must renew a 30-second . Three misses means trouble.
- 2Agree by majority
Standby in zone B + witness in zone C = 2 of 3 votes, a . An isolated node can never win alone.
- 3Fence the old primary
Its lease has expired, so it stops accepting writes, even if it's alive behind a broken link. That prevents : two leaders accepting different writes.
- 4Promote the standby
Only a synchronous standby, which has every confirmed booking. One writer and a waiting standby is active-passive.
- 5Re-route clients
A connection proxy points at the new primary; apps reconnect in seconds. A DNS name with a 300 s TTL would be the slowest phase.
- 6Restore redundancy
We're on one node now. Build a new standby, and keep alerting until it exists.
The app servers are the easy half: , so they run active-active, all serving, and a dead one is just skipped. But size them N+1: losing a zone must leave enough capacity.
Watch a decision being made
4 minYou draft StayHub's failover design, Lina reviews
Make losing a server, a database node or a whole zone cost seconds, not 41 minutes. Today: 4 app servers in zones A and B at ~75% CPU at peak; primary in A, sync standby in B, manual promotion.
- 1Size N+1 per zone
Peak needs 3 servers' worth; losing zone A leaves 2 needing 150%. Move to 3 zones × 2 = 6: losing any zone leaves 4 at ~75%.
Why: Lina: "two of everything isn't enough if the survivors can't carry the load."
- 2Majority, lease, in-sync only
Witness in zone C, 2 of 3 votes, 30 s lease; promote only while the standby is in sync.
yamlttl: 30 # leader lease loop_wait: 10 # renew / check every 10 s synchronous_mode: true # only a sync standby can be promotedWhy: A cut-off node can't elect itself, the old primary steps down alone, and no confirmed booking is lost.
- 3A stable endpoint for apps
Apps connect through a proxy the failover manager updates, not the DNS name.
Why: Detection and promotion take ~40 s; a 300 s DNS cache would dwarf them.
- 4Drill it every quarter
Kill the primary in a drill, time it end to end (~30 s + ~10 s + ~5 s: under a minute), count bookings before and after.
Why: Untested failover fails when you need it.
In real life
3 minNetflix: active-active, shift the load
- Has publicly described serving from several regions at once and moving traffic off a failing one
- Works because most services are stateless and its data stores accept writes in several places
- No promotion step
Fits when copies can accept writes without conflicts that matter.
GitHub: one writer, automatic promotion
- Runs MySQL with a single primary and automated failover tooling
- Its public October 2018 report describes a 43-second network interruption leading to a cross-site promotion
- Writes landed on both sides; repair meant about a day of degraded service
Single-writer is safer for data like ours, if promotion is fenced and scoped.
What goes wrong
3 minA broken link looks exactly like a dead peer: split brain.
An odd number of voters, a majority, fencing before promotion.
Survivors at 150% can't absorb a lost zone.
Stay under ~70–80% after losing a zone.
Standbys drift, TTLs are longer than assumed, pools hold dead connections.
Regular timed drills; fix what breaks.
Trade-offs
3 min| Active-passive | Active-active | |
|---|---|---|
| Writers | One: no conflicts | Several: conflicts to handle |
| Spare capacity | Idle standby | All used, each with headroom |
| On failure | Seconds to promote, fenced | Load just shifts |
| Fits | Databases, inventory, payments | Stateless tiers, data built for it |
- Karim · CTOWe pay for a full-size standby that does nothing all day. Let both zones take bookings.
- Lina · Senior engineer, your mentorTwo writers for room inventory means two guests can book the last room in the same second.
- Karim · CTOThen lock rooms per zone, or fix conflicts afterwards. Idle hardware is money burning.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Sizing for a lost zone: 6 servers across 3 zones is a cost you choose, not something to trim later "because they're idle".
- Keeping the standby synchronous (~1.5 ms per write) so no confirmed booking is lost.
- Who may promote: 2 of 3 votes with a witness in zone C and a 30 s lease. Never "whoever can't see the primary".
Plan only, no changes yet: we're replacing the manual database promotion that cost us 41 minutes this week. Target: sync standby in zone B, witness in zone C, promotion needs 2 of 3 votes, lease TTL 30 s, the old primary fenced before promotion, apps connecting through the connection proxy, not the DNS name (300 s TTL). List what changes, step by step, with rollback. Then write a staging drill: stop the primary, measure detection, promotion and re-route times, and check no booking confirmed before the stop is missing afterwards.
- The order: detect, agree (2 of 3), fence, promote, re-route, rebuild. Fencing missing or after promotion: reject.
- The drill timeline: ~30 s + ~10 s + ~5 s, under a minute.
- No lost booking proven by a count before and after, not "looks fine".
- Automatic promotion with two nodes and no witness.
- Switching the standby to async "for performance" without saying it can lose confirmed bookings.
- Adding every dependency to the readiness check "to be thorough".
Check yourself
2 minNext episode: a migration sets every hotel's price to €0, the standby copies it perfectly, and you lead the recovery.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You draft StayHub's failover design, Lina reviews
Make sure losing a server, a database node or a whole zone in Frankfurt costs StayHub seconds, not 41 minutes. Today: 4 app servers in zones A and B at ~75% CPU at peak; a primary in A with a synchronous standby in B (set up in the scalability module) and manual promotion.
- 1List the single points of failure
The DB primary (manual promotion), and the app tier: peak needs 3 servers' worth of capacity, and losing zone A leaves 2 servers needing 150%.
Why: Lina's review note: "Two of everything isn't enough if the survivors can't carry the load."
- 2Size N+1 per zone
Move to 3 zones × 2 servers = 6. Losing any zone leaves 4 servers at ~75% CPU, the same as today's peak.
Why: N+1 is about the failure you plan to survive. Surviving a zone means each zone's share must be spare.
- 3Promote only an in-sync standby
The zone-B standby is already synchronous. Tell the failover manager to promote it only while it's in sync, never a copy that has fallen behind.
Why: A standby that is behind (or async) may not have the last bookings a guest saw confirmed. We already pay ~1.5 ms per write for sync; losing confirmed bookings would cost far more.
- 4Add a majority and fencing
A witness in zone C. Promotion needs 2 of 3 votes. The primary holds a 30 s lease; once it expires, it refuses writes.
yamlttl: 30 # leader lease: not renewed within 30 s → leadership lost loop_wait: 10 # renew / check every 10 s synchronous_mode: true # only a synchronous standby can be promoted # coordination store: 3 members across zones A, B, CWhy: Majority stops a cut-off node from electing itself; the lease makes the old primary step down on its own.
- 5Give apps a stable endpoint
Apps connect through a database proxy that the failover manager updates, instead of a DNS name with a 300 s TTL.
Why: Detection and promotion take ~40 s. A 300 s DNS cache would make re-routing the slowest phase by far.
- 6Drill it and restore redundancy
Kill the primary in a drill every quarter, time it end to end, and automatically build a new standby afterwards.
Why: Untested failover fails when you need it. And after a failover we're on one node: a second failure would be a full outage.
| Failover phase | Typical duration | What controls it |
|---|---|---|
| Detection | 5–30 s | Check interval × failure threshold, lease TTL |
| Decision (majority agrees) | < 1–5 s | Coordination store latency |
| Fencing old leader | < 1–10 s | Lease expiry, network or storage isolation |
| Promotion | 1–10 s | Replaying the remaining log on the standby |
| Re-routing clients | seconds to minutes | Proxy or virtual IP (fast) vs DNS TTL (slow) |
Interactive simulation
Visual diagram
Active-active app tier across three zones, active-passive database with a witness. Run the walkthrough to see a primary failure.
Analogy
- Flying a planeCan still fly with one engine outRunning a serviceOne extra server: losing one is fine
- Flying a planeCaptain flies, co-pilot ready to take overRunning a serviceActive-passive: a standby waits
- Flying a planeCo-pilot cross-checks the instrumentsRunning a serviceHealth checks watch each copy
- Flying a planeClear spoken handover: "Your controls"Running a serviceSafe failover with a single leader
- Flying a planeBoth pilots steering different waysRunning a serviceSplit brain: two leaders, conflicting data
Where the analogy breaks: pilots can see and talk to each other. Servers often can't tell if their partner is dead or just unreachable, which is exactly why split brain happens.
Check your understanding
- 1.Primary misses heartbeats / fails to renew its lease
- 2.A new standby is created to restore redundancy
- 3.Old primary is fenced so it can't accept writes
- 4.A majority of voters agrees the primary is gone
- 5.The most up-to-date (synchronous) standby is promoted
- 6.The database endpoint is switched and clients reconnect
Practice ladder
Trace a 15-second network split
Scenario Warm-upStayHub's reviews service has two database nodes and no witness: a primary in zone A and a standby in zone B. Rule: the standby promotes itself if it can't reach the primary for 10 seconds.
At 11:00:00 the network link between zones A and B drops for 15 seconds. App servers in zone A can still reach the primary; app servers in zone B can only reach the standby.
Trace what happens second by second until 11:00:30, and what state the data is in afterwards.
Fix the order of Omar's runbook
Design PracticeOmar wrote the automatic failover steps for the payments database, but one is missing and two are swapped. Put them in the right order and add the missing step.
- A. Promote the standby to primary
- B. Point the database proxy at the new primary
- C. Detect: the primary missed 3 health checks and its lease renewals
- D. The failover manager and the standby agree (2 of 3 votes) the primary is gone
- E. Build a new standby and alert until it's in sync
- F. ___
Size the search fleet for a lost zone
Estimation PracticeThe search API needs 8 app servers at peak (each sized to run at ~70% CPU). Each server costs about €200 a month. StayHub must survive losing a whole zone at peak without slowing down.
- How many servers, and what monthly cost, with 2 zones? With 3 zones?
- Which would you pick, and what else must be true for the zone spread to actually help?
Sync standby for the reviews database?
Decision StretchThe reviews database (guest reviews of hotels) has an async standby and manual promotion. Two options:
- A. Same as bookings: sync standby, witness in zone C, automatic fenced failover. About €700/month more and ~1.5 ms added to each review write.
- B. Keep async, but automate promotion with a witness and fencing. Failover in under a minute; may lose the last few seconds of reviews.
Reviews are written about 20 times a minute. A lost review means a guest re-types it (we can email them). Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain how you'd make a booking system survive the loss of a server, a database node and a whole availability zone, and what could go wrong during failover.