Disaster recovery: backups, RPO/RTO and multi-region strategies
Tuesday, 14:08: a migration sets the price of all 9,000 hotels to €0. By 14:17 guests have booked 412 rooms for free, and the synchronous standby copied the damage in under a millisecond.
Spares save you from a dead server. Disaster recovery saves you from a lost region, deleted data, or an attacker.
- Problem
A hotel booking platform must keep taking reservations even if a whole cloud region goes dark or someone runs `DELETE` without a `WHERE`.
- Why the simple system fails
Replicas and multi-AZ protect against hardware failure, but faithfully replicate a bad deletion, and they all live in one region. Untested backups often turn out to be incomplete or too slow to restore.
- Engineering concept
Decide how much data you may lose (RPO) and how long you may be down (RTO), then buy exactly the recovery strategy that meets them, and rehearse it.
- Technology
Snapshots, point-in-time recovery from the write-ahead log, cross-region replication, immutable backups, DNS failover, infrastructure as code; strategies: backup-restore, pilot light, warm standby, active-active.
- Trade-offs
Each step up the ladder cuts RPO/RTO by roughly an order of magnitude and multiplies cost and operational complexity. Active-active also forces you to solve multi-region writes.
The problem
2 min- Omar · On-call SREMigration 0412 ran at 14:08 and set price = 0 for all 9,000 hotels. 412 rooms booked at €0.
- Omar · On-call SRELet's fail over to the standby in zone B. It's healthy.
- Lina · Senior engineer, your mentorYou lead this one. First: which copy still has the right prices?
The big idea
4 minRedundancy protects you from failures. Backups protect you from mistakes, lost regions and attackers. A replica is never a backup, and a backup you haven't restored is a hope.
Prices were back at 14:55. Lina's note: now ask the same question for a lost region. Our backups sit in Frankfurt, in the same account as production.
How it works
5 min- 1Stop the bleeding
Pause booking writes and block the migration from re-running. Every minute adds more €0 bookings.
- 2Restore next to production
: the 02:00 snapshot plus the logged changes (WAL) replayed to 14:07:59, into a fresh instance. Never over production.
- 3Copy only what broke
Copy the prices table back; bookings since 14:08 stay. Then decide case by case for the 412 €0 bookings.
- 4Name the targets
: how much data you may lose, in time. WAL every 5 minutes gives about 5. RTO: how long you may be down. The chain's contract says 1 hour for bookings.
- 5The DR ladder
Backup and restore → (database replicated, servers off) → (small working copy) → active-active. Each faster to recover, and more expensive.
- 6Out of reach
Backups in a separate account, for 30 days. A region away is not a permission away.
Watch a decision being made
4 minYou write StayHub's DR plan, Lina checks it
Plan for losing Frankfurt, a destroyed table, or a stolen admin key. An hour of checkout outage costs about €20,000.
- 1Set RPO and RTO per tier
Bookings: 5 min and 1 h. Extranet: 1 h and 8 h. Analytics: rebuild from source within a week.
Why: Lina: "one target for everything would buy the most expensive strategy for systems that don't need it."
- 2Measure what you have
Run the restore drill: 308 minutes (download 190, WAL replay 95). The design doc said 60.
bashrestore-db --from s3://bookings-backups-dr --target drill-$(date +%F) \ --point-in-time "$TARGET_TIME" psql "$DRILL_URL" -c "SELECT max(created_at) FROM bookings;" # gap = achieved RPOWhy: Only a drill tells you the real number.
- 3Pick the cheapest rung that meets it
Bookings: warm standby in a second EU region (~€6,500/month, RTO ~20 min). Extranet: backup and restore, snapshots every 6 h.
Why: Pilot light (~€2,500) needs ~2 h: misses the 1-hour RTO. Active-active is ~€25,000 plus months for an RTO we don't need.
- 4Locked backups, a deliberate switch
Backups to a separate account with object lock. DNS TTL to 60 s. A human declares the disaster; then one command fences, promotes, scales and switches. Drill it twice a year.
Why: Automatic cross-region failover on one health check risks data loss on a blip.
In real life
3 minGitLab.com, 2017: untested backups
- Publicly documented an incident where a production database directory was deleted by mistake
- Several backup and replication methods turned out not to be working
- Restored from a copy about 6 hours old and lost that window of some data
Backups existed on paper; nobody had proven a restore.
Netflix: region evacuation practised
- Has publicly described serving from several regions and shifting all traffic off one
- Practises these evacuations on purpose
- Pays for capacity in several regions all the time
Near-zero RTO, at the top rung's price.
What goes wrong
3 minA bad UPDATE or ransomware reaches every replica almost instantly.
Point-in-time backups with retention, separate from replication.
Restores fail for boring reasons: permissions, keys, a three-hour download.
Automated drills that record achieved RPO and RTO.
Multi-region writes need conflict handling and roughly double the cost.
The cheapest rung that meets each tier's targets.
Trade-offs
3 min| Strategy | RTO | Monthly (StayHub) | Catch |
|---|---|---|---|
| Backup and restore | Hours (308 min measured) | Lowest | Rebuilding under stress |
| Pilot light | ~2 h | ~€2,500 | Unused paths rot |
| Warm standby | ~20 min | ~€6,500 | Async: last seconds lost |
| Active-active | ~0 | ~€25,000 + months | Multi-region write bugs |
- Maya · Product managerGuests should never see StayHub down. Let's run both regions live, like the big players.
- Omar · On-call SREThen two regions write room inventory, every overbooking bug becomes cross-region, and it's ~€25,000 a month.
- Maya · Product managerA warm standby still means 20 minutes down if Frankfurt goes. Thousands of guests.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- RPO and RTO per system: bookings 5 min and 1 h, extranet 1 h and 8 h, analytics a week. From the business, not from what's easy to build.
- Which rung you pay for: warm standby (~€6,500/month), pilot light (~€2,500, ~2 h RTO) or active-active (~€25,000 plus months).
- Where backups live, and who declares a regional disaster: a human, never a single health check.
Write a restore drill for the booking database, staging only, never production. Goal: measure our real RTO for a point-in-time restore; the design doc says 60 minutes and nobody has tested it. Restore the latest snapshot plus WAL to a target time into a fresh instance next to the existing one, verify row counts and a sample of prices against the source, then tear it down. Log a timestamp per phase. Don't change backup settings or retention. Put the script and a runbook in ops/dr/ and show me the plan before running anything.
- The drill restores next to the database, into a new instance, never over it.
- Read the timestamps per phase: believe the stopwatch (our first took 308 minutes), not the doc.
- Verification uses counts and spot checks against a known time, not "the database started"; the teardown can't touch production by name.
- It calls the replica or the zone standby "the backup".
- It proposes restoring the nightly snapshot over production to fix one table.
- It calls backups in another region of the same account safe from a stolen key.
Check yourself
2 minNext episode: checkout is slow for one guest in fifty, every dashboard looks normal, and you need to see inside the system instead of guessing.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You write StayHub's DR plan, Lina checks it
Turn "don't lose data, don't be down long" into a plan for losing Frankfurt, a destroyed table, or a stolen admin key. An hour of checkout outage costs about €20,000.
- 1Set RPO and RTO per tier
Bookings: RPO 5 min, RTO 1 h (in the chain's contract). Extranet: RPO 1 h, RTO 8 h. Analytics: rebuild from source within a week.
Why: Lina's check: "Good. One target for everything would buy the most expensive strategy for systems that don't need it."
- 2Measure what you have
Run the restore drill. Result: 308 minutes (download 190, WAL replay 95). Backups live in Frankfurt, same account.
bashrestore-db --from s3://bookings-backups-dr --target drill-$(date +%F) \ --point-in-time "$TARGET_TIME" psql "$DRILL_URL" -c "SELECT max(created_at) FROM bookings;" # gap = achieved RPO BASE_URL=$DRILL_APP_URL pnpm test:smokeWhy: The design doc said 60 minutes. Only a drill tells you the real number.
- 3Pick the cheapest rung that meets it
Bookings: warm standby in a second EU region (async replica + 2 small app servers, ~€6,500/month, RTO ~20 min). Extranet: backup and restore, with snapshots every 6 h.
Why: Pilot light (~€2,500/month) needs ~2 h to start and scale: it misses the 1-hour RTO. Active-active would be ~€25,000/month plus months of work for an RTO we don't need.
- 4Put backups out of reach
Copy snapshots and WAL to a separate backup account, with object lock (no deletes for 30 days). Production can write there, never delete.
Why: A region away is not a permission away. One stolen key must not reach both production and its backups.
- 5Plan the switch
Lower the DNS TTL to 60 s. Declaring a regional disaster is a human decision; after that, one command fences the old region, promotes the replica, scales the app servers and switches DNS. Check the payment provider accepts calls from the new region's IPs.
Why: Automatic cross-region failover on one health check risks data loss on a blip. A rehearsed, one-command switch is fast and deliberate.
- 6Drill and record
Game day twice a year for the region switch, restore drill every quarter. Write down achieved RPO and RTO each time.
Why: Data grows, configs drift, people change. The numbers you measured last year are already wrong.
| Strategy | Typical RPO | Typical RTO | Standby cost (vs prod) | What runs in the DR region |
|---|---|---|---|---|
| Backup and restore | Minutes to 24 h | Hours to a day | ~2–10% | Nothing; backups in object storage |
| Pilot light | Seconds to minutes | Tens of minutes to hours | ~10–20% | Replicated database; app servers off or minimal |
| Warm standby | Seconds | Minutes | ~25–50% | Replicated database + small working app fleet |
| Active-active | ~0 to seconds | ~0 (traffic shifts) | ~100%+ plus engineering | Full production serving live traffic |
Interactive simulation
Visual diagram
A warm-standby DR setup for a booking platform. Follow the walkthrough from normal operation to a regional failover.
Analogy
- A hotel chain after a fireRegister photocopied nightly, kept in another cityYour system after a disasterBackup and restore
- A hotel chain after a fireEmpty sister building, register copied there constantlyYour system after a disasterPilot light
- A hotel chain after a fireSister building with a skeleton crewYour system after a disasterWarm standby
- A hotel chain after a fireTwo full hotels, guests in bothYour system after a disasterActive-active regions
- A hotel chain after a fireFire drill with a stopwatchYour system after a disasterA real restore test
Where the analogy breaks: a fire is obvious. A data disaster can be silent, like a bug quietly corrupting prices for a week. That's why you need backups from many points in time, not just the latest copy.
Check your understanding
Practice ladder
Trace a region loss at 16:42
Scenario Warm-upBefore the warm standby exists: the booking database is in Frankfurt. A snapshot is taken nightly at 02:00 and copied to the second region. WAL is shipped to the second region every 5 minutes, at :00, :05, :10…
At 16:42 the Frankfurt region goes dark. The last WAL shipment completed at 16:40.
Trace: up to which moment can you recover bookings, how much data is lost (and what's the worst case on another day), and which two phases will dominate the restore time?
Fill the gaps in the region-switch runbook
Design PracticeYour runbook for switching bookings to the warm standby region has two gaps. Fill in 2 and 5.
- Declare: the incident commander declares a regional disaster, using signals from several places (not one health check).
- ___
- Promote the async replica in the second region to primary.
- Scale the app servers there from 2 to 6.
- ___
- Check partners: the payment provider must accept calls from the new region's IP addresses.
- Communicate on the status page, then plan failback.
Can the extranet meet a 4-hour RTO?
Estimation PracticeThe hotel extranet's database is 400 GB. Plan: backup and restore into the second region. Targets: RPO 1 h, RTO 4 h (a large chain is asking for 4 h instead of 8).
- Snapshot nightly at 02:00, copied to the second region; WAL shipped every 5 minutes.
- Download speed into a new instance: about 200 MB/s.
- WAL replay: 1 hour of changes takes about 8 minutes to replay.
- Rebuilding servers, secrets and DNS with Terraform: about 45 minutes.
- Verification and DNS switch: about 20 minutes.
A bad case: the region dies at 23:00, 21 hours after the last snapshot. Does the plan meet RPO 1 h and RTO 4 h? If not, what's the cheapest change?
Pilot light or warm standby for bookings?
Decision StretchFor bookings, two options in the second region:
- Pilot light: replicated database, app servers off. €2,500/month. Measured switch in a game day: ~2 hours (start and warm up servers, scale, DNS).
- Warm standby: replicated database + 2 small app servers serving synthetic traffic. €6,500/month. Measured switch: ~20 minutes.
Facts: the hotel chain's contract says RTO 1 h, with a €40,000 penalty per breach. An hour of checkout outage costs about €20,000. A full regional outage is rare, maybe once in several years.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain to a product manager how you would choose a disaster-recovery strategy for the booking API, and why you wouldn't just pick the safest option.