Skip to content
Episode 54 · One zone goes dark

Regions and availability zones

Thursday, 14:20: one of Frankfurt's three data-centre zones loses power. Two-thirds of StayHub's servers are perfectly healthy, yet bookings stop for 47 minutes.

"The cloud" is real buildings in real places. Where yours are decides speed, what failures you survive, and which laws apply.

30 min intermediateCloud regions & availability zonesData residency & GDPR constraintsMulti-AZ vs multi-region deploymentLatencyRead replicas
  1. Problem

    A hotel booking platform must stay up when a data centre loses power, answer European guests fast, and keep EU guests' personal data where the law allows.

  2. Why the simple system fails

    Everything in one data centre: one fire, flood or power fault takes the whole business down, and moving data later is a painful migration.

  3. Engineering concept

    Clouds are split into regions (geography) made of availability zones (isolated data centres). Spread across zones for failures; choose regions for latency and law.

  4. Technology

    Regions, availability zones, multi-AZ load balancers and databases, cross-region replication, DNS failover.

  5. Trade-offs

    Each step of geographic spread buys resilience and latency at the price of cost, consistency and operational complexity.

01

The problem

2 min
Thursday, 14:20. The provider's status page turns orange.
  1. Omar · On-call SRE
    Power loss in zone eu-central-1a. App VMs in zones b and c: healthy. Bookings: zero. Logins failing.
  2. Karim · CTO
    We just rebuilt across three zones. How is one zone taking us down?
  3. You · Engineer at StayHub
    Where do the database and the session cache run?
02

The big idea

4 min
Healthy servers, no bookings
The big idea

Multi-AZ means every tier, especially the ones holding data.

The database was restored by hand into zone b: 47 minutes, about 800 bookings lost. With a synchronous standby in another zone, the same failure costs one to two minutes and no committed data.

03

How it works

5 min
Regions, zones, and what each one survives
  1. 1
    Region

    A is a geographic area (Frankfurt, Paris, Singapore) where a provider runs several data centres. Chosen for latency to users and for the law. Regions are far apart and independent.

  2. 2
    Availability zones

    An availability zone is a separate data centre inside a region, with its own power and network, about 1–2 ms from its neighbours: close enough to wait for, far enough not to fail together.

  3. 3
    Multi-AZ: a standby that keeps up

    : each write is stored on the primary and a standby in another zone before it's confirmed. If the zone dies, the standby is promoted in about 60–120 s.

  4. 4
    RPO and RTO

    RPO is how much recent data you can lose; RTO how long you can be down. Multi-AZ zone failure: RPO ≈ 0, RTO ≈ 2 minutes.

  5. 5
    Multi-region: far, so async

    survives losing a whole region. At 10–160 ms per round trip, replication must be asynchronous, accepting a few seconds of possible loss.

  6. 6
    Region choice is a legal choice

    rules like decide which regions may hold EU guests' data. Backups and logs count.

04

Watch a decision being made

4 min
Worked example

You make every tier survive a zone; Lina checks

A full Frankfurt zone failure must cost no committed bookings and at most a couple of minutes of errors.

  1. 1List every stateful piece

    PostgreSQL (zone a only), Redis sessions (zone a only), object storage (regional), Kafka (managed, 3 zones).

    Why: Outages hide in the pieces that hold data. Lina's check: confirm managed services' zone settings, don't assume.

  2. 2Give each a standby in another zone

    PostgreSQL multi-AZ with a synchronous standby; Redis replica with automatic failover.

    hcl
    resource "aws_db_instance" "bookings" { multi_az = true }
resource "aws_elasticache_replication_group" "sessions" {
  automatic_failover_enabled = true
  multi_az_enabled           = true
}

    Why: 1–2 ms per write buys RPO ≈ 0 for a zone failure.

  3. 3Size for losing a zone

    9 app VMs, each zone sized for 50% of peak, so any two carry 100%.

    Why: The survivors absorb the lost zone's traffic at the busiest hour.

  4. 4Write RPO/RTO down, then drill

    Zone: RPO ≈ 0, RTO ≈ 2 min. Region: restore from Paris backups, hours. Force a failover every quarter and time it.

    Why: Apps that cache connections or DNS can fail to follow the new primary; only a real failover shows it.

The whole thing in one line: Every single-zone piece got a synchronous standby, the app was sized to lose a zone, and the promise became numbers you drill.
05

In real life

3 min
Drain a zone versus evacuate a region

Slack: zone-level isolation

  • Has publicly described reworking its architecture so traffic can be drained from one availability zone
  • A sick zone shouldn't hurt the rest of the region
  • Big effort, within one region's latency

Fits when zone problems are the outages you actually see.

Netflix: region-level evacuation

  • Has publicly described running in several AWS regions and practising moving all traffic out of one
  • Survives a whole region failing
  • Needs cross-region data and constant rehearsal

Fits a global product where a region outage is unacceptable.

06

What goes wrong

3 min
Multi-AZ app servers, single-AZ database

When the database's zone fails, every app server is healthy and useless.

Instead

Multi-AZ on every stateful service, with a tested failover.

Sizing each zone for exactly its share

Losing a zone pushes the survivors past 100%.

Instead

Size so the remaining zones carry the peak.

Picking an EU region and calling GDPR done

Backups, logs and vendors quietly move data elsewhere.

Instead

DR copies in the same jurisdiction (Paris), and a data map.

07

Trade-offs

3 min
SetupSurvivesYou pay
Single-AZA server crash (if ≥ 2)A zone outage is a full outage
Multi-AZA data centre, in minutes~1.3–2× for data tiers
Multi-region active-passiveA region, after failoverIdle capacity; seconds of data loss
Multi-region active-activeA region, near-instantlyWrite conflicts; heavy engineering
Friday, 11:00. The postmortem.
  1. Karim · CTO
    47 minutes, 800 bookings. Let's run active-active in Frankfurt and Paris.
  2. Lina · Senior engineer, your mentor
    Thursday was a zone failure. Multi-AZ fixes it for about €6,000 a month. Active-active is months: who owns a booking when both regions take writes?
  3. Karim · CTO
    And when the whole region goes down?
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • Which failures StayHub must survive: a zone automatically; a region from Paris backups and a written plan.
  • Which stateful pieces must be multi-AZ: bookings PostgreSQL and the session Redis.
  • Where every copy of EU guest data may live: Frankfurt and Paris yes; Singapore no.
Tell Claude Code
In our infrastructure code, switch the Frankfurt bookings PostgreSQL to multi-AZ with a synchronous standby in another zone, and the same for the Redis session store. Don't change instance sizes, Singapore or backup destinations. The app must connect by the provider's endpoint name, never an IP, and retry for up to two minutes during failover: show me today's client settings. Show the plan first, with the monthly cost increase and any restart. Then write a staging failover runbook with what to measure.
Check before you accept
  • Both the database and Redis get a standby in another zone; nothing else changes size or region.
  • The app connects by endpoint name and retries: Claude shows the exact setting.
  • In the staging failover, bookings fail for about 1–2 minutes, not 47.
Red flags
  • Synchronous replication to another region.
  • EU backups or logs copied to Singapore or the US "for disaster recovery".
  • "Highly available" because app servers span 3 zones while the database is in one.
09

Check yourself

2 min
Q1/3Scenario
Your app runs 4 servers across zones a and b behind a load balancer, but the PostgreSQL instance is single-AZ in zone a. Zone a loses power. What happens?
How confident are you?

Next episode: Karim forwards a sales pitch from a second cloud provider promising 30% savings, and you have to decide what it would really take to move.

Finished?

Mark it complete to update your skill map.

Next: The cloud provider map: AWS, Azure and GCP
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

A database in two zones
Worked example

You make every tier survive a zone; Lina checks

Make sure a full zone failure in Frankfurt costs StayHub no committed bookings and at most a couple of minutes of errors.

  1. 1List every stateful piece

    Bookings PostgreSQL (zone a only), Redis sessions (zone a only), object storage (regional, already multi-zone), Kafka (managed, 3 zones).

    Why: Stateless servers are easy to spread; outages hide in the pieces that hold data. Lina's check: include managed services and confirm their zone settings, don't assume.

  2. 2Give each a standby in another zone

    PostgreSQL: multi-AZ with a synchronous standby. Redis: a replica in zone b with automatic failover.

    Why: Synchronous across zones costs 1–2 ms per write and means a zone failure loses nothing that was committed (RPO ≈ 0).

  3. 3Size for losing a zone

    9 app VMs, 3 per zone, each zone sized for 50% of peak, so any two zones carry 100%.

    Why: Surviving a zone means the remaining zones must absorb its traffic, at the busiest hour, without falling over.

  4. 4Write down RPO and RTO

    Zone failure: RPO ≈ 0, RTO ≈ 2 minutes. Region loss: restore from Paris backups, RPO ≤ 5 minutes (log shipping), RTO a few hours.

    Why: Numbers turn "we're resilient" into a promise you can check, and show the business what a region loss would still cost.

  5. 5Drill it

    Every quarter, force a database failover in a quiet hour and time it end to end: errors, reconnects, alerts.

    Why: Apps that cache connections or DNS can fail to follow the new primary. Only a real failover reveals that.

The whole thing in one line: You found the single-zone pieces, gave each a synchronous standby in another zone, sized the app so two zones carry the peak, wrote down RPO and RTO for zone and region loss, and scheduled drills. Lina's only addition: Kafka's "3 zones" needed checking, not assuming.
GoalSingle-AZMulti-AZMulti-region (active-passive)Multi-region (active-active)
Survives a server crashIf ≥ 2 serversYesYesYes
Survives losing a data centreNoYes, minutesYesYes
Survives losing a regionNoNoYes, after failoverYes, near-instant
Typical RPO in a disasterLast backup~0 for a zoneSeconds (lag)Seconds, plus conflicts
Relative cost1×~1.3–2×~1.5–2.5×~2–3× plus heavy engineering
Illustrative multipliers. The real cost of active-active is mostly engineering time.
11

Visual diagram

Click a component

A multi-AZ booking platform in an EU region, with an optional standby region for disaster recovery.

12

Analogy

  • A bankA city where the bank operatesThe cloudA region
  • A bankThree buildings, each with its own generatorThe cloudAvailability zones
  • A bankLedgers copied in two buildingsThe cloudMulti-AZ
  • A bankA second branch in another cityThe cloudMulti-region
  • A bank"Records may not leave the country"The cloudData residency rules

Where the analogy breaks: a bank clerk can phone another city and wait. Software syncing across regions waits on every write, so most systems copy data across regions with a delay, and can lose the last few changes in a disaster.

13

Check your understanding

Q1/3Decision
Match each need to the placement that addresses it:
Survive a single data centre losing power
Fast pages for a large user base in Australia
Contract requires recovery from a region-wide outage within 1 hour
German health data must not leave the jurisdiction
How confident are you?
14

Practice ladder

Step 1 · Trace

Trace a zone loss at peak

Scenario Warm-up

Before your fix, StayHub's API ran 6 VMs, 2 per zone, at 70% CPU each during the evening peak. The database was already multi-AZ.

Zone c loses power at 20:00 (peak).

  1. What CPU load do the remaining VMs face?
  2. What happens over the next minutes if autoscaling takes 5 minutes to add VMs?
  3. How many VMs per zone would let two zones carry the peak at no more than about 70% CPU?
Step 2 · Complete

Fill in the recovery table

Design Practice

Lina's table states what StayHub can lose (RPO) and how long it's down (RTO) in each failure, after your fixes. Fill in the blanks.

FailureProtectionRPORTO
One app VM crashes9 VMs, load balancer0seconds
Zone a loses powerMulti-AZ DB (sync), Redis replica, 3 zones sized at 50%______
Someone runs DELETE FROM bookings by mistakePoint-in-time recovery, 14 days______
The whole Frankfurt region is unavailableBackups + log shipping to Paris≤ 5 min___
Step 3 · Solve

Place StayHub for American guests

Design Practice

20% of StayHub's guests are now in the US, and search feels slow there (Frankfurt ↔ US East ~90 ms per round trip, several round trips per page). Leadership wants: faster search in the US, no loss of committed bookings if a data centre fails, EU guests' personal data kept in the EU, and a team of 18 engineers.

Propose where each piece runs (search, hotel content, bookings, guest profiles, backups) and say what you'd not do yet.

Step 4 · Decide

Synchronous replication to Paris?

Decision Stretch

After the BergHotels questionnaire, Karim asks: "Can we promise zero data loss even if all of Frankfurt fails? Make the Paris copy synchronous."

Facts: Frankfurt ↔ Paris round trip ~10 ms. A checkout makes 5 sequential writes. At peak, about 1 checkout per second (~1,000 bookings an hour by day, more at sale peaks). Today Paris receives an async log stream, typically under 5 seconds behind.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

15

Explain it yourself

Explain to a product manager the difference between multi-AZ and multi-region, and when you'd recommend each.

0 words