The nines: how much availability do you need?
Tuesday, 9:30: the hotel chain wants 99.99% on checkout, which allows 4.4 minutes of downtime a month. StayHub's last bad deploy took 40 minutes to roll back, and Karim wants to know what it would take.
99.9% sounds almost perfect. Turn it into minutes of downtime and you'll see why each extra 9 changes everything.
- Problem
A customer asks for “five nines”. Your team wants to know what that means in practice, whether you can achieve it, and what it would cost.
- Why the simple system fails
Percentages hide the scale: 99.9% sounds close to 99.999%, but one allows almost 9 hours of downtime a year and the other about 5 minutes. And every component you add in a chain lowers the total.
- Engineering concept
Convert targets into downtime budgets, multiply availabilities along a request path, use redundancy to raise weak links, and price each nine against what an outage actually costs.
- Technology
Availability arithmetic (series and parallel), MTBF/MTTR, redundancy across zones and regions, automated failover, on-call.
- Trade-offs
Each extra nine typically multiplies infrastructure and operational cost and constrains how fast you can change things. Most products need three nines, not five.
The problem
2 min- Karim · CTOWe do 99.9%. They want 99.99%. That's 0.09% apart. Can we just try harder?
- You · Engineer at StayHubIt does sound close.
- Lina · Senior engineer, your mentorPercentages hide the scale. Guess first, then we'll turn them into minutes.
The big idea
4 minIn a chain, availability multiplies down; a spare copy only helps the box it copies. Find the weakest link before you buy anything.
44 minutes a month lets Omar wake up and roll back. 4.4 minutes does not: at , only automation reacts fast enough.
How it works
5 min- 1Turn % into minutes
Allowed downtime = (1 − availability) × period. 99.99% of a month = 4.4 min.
- 2Multiply the series
Parts in series must all work: DNS × LB × app × DB × payments = 0.9999 × 0.9999 × 0.999 × 0.9995 × 0.9995 ≈ 99.78%, about 19 hours a year.
- 3Combine the spares
Copies in parallel, where any one is enough: 1 − (chance all fail at once). Two app servers in different zones: 1 − 0.001² ≈ 99.9999%. Path: 99.88%.
- 4Find the weakest link
Now the database and the payment provider (99.95% each) set the ceiling. Checkout can't beat 99.95% while every booking needs that provider.
- 5Check independence
Parallel maths assumes copies fail separately. A shared deploy, config or makes them fail together.
- 6Count recovery time
(mean time to recovery) includes noticing. One incident a month at 40 minutes: 99.91%. At 5 minutes: ~99.99%.
Watch a decision being made
4 minYou and Lina price the fourth nine
Karim needs a yes or no on 99.99% for checkout, and a cost. You have last year's incidents and the SLAs of everything on the path. You multiply; Lina checks.
- 1Multiply the critical path
Series of everything, parallel for the app tier.
tsconst series = (...a: number[]) => a.reduce((acc, x) => acc * x, 1); const parallel = (a: number, n: number) => 1 - Math.pow(1 - a, n); series(0.9999, 0.9999, parallel(0.999, 2), 0.9995, 0.9995); // ~0.9988Why: The ceiling is the whole path, not the best component: 99.88% is below 99.9% on paper.
- 2Check what really broke
Last year ~526 minutes down: bad deploys ~368 (70%), payment provider ~105 (20%), the rest ~53. About one incident a month, 40 minutes each.
Why: The formula assumes random hardware failures. Real downtime is mostly our own changes, which hit every copy at once.
- 3Pick the cheapest levers
1) Canary deploys + automatic rollback: MTTR 40 → ~5 min. 2) A "pay at the hotel" fallback so checkout survives provider outages. 3) Automatic database failover, measured.
Why: Each attacks a real slice of the 526 minutes. A second region would attack the 10% "everything else" at the highest price.
- 4Answer with conditions
"Not today. After canaries and the fallback we can offer 99.95%, and reconsider 99.99% after six months there."
Why: Sign only what the maths and your track record support.
In real life
3 minAWS guidance: spread copies across zones
- Its public Well-Architected guidance recommends running across several availability zones
- One zone's failure leaves the others
- Raises the boxes you control by running spares
Works when the weak link is your own hardware or zone, and failures are independent.
Netflix: degrade gracefully, test failure
- Has publicly described fallbacks: if personal recommendations fail, show something generic
- Built Chaos Monkey to kill servers on purpose
- Raises what users feel without making every dependency perfect
Works when you can't control a dependency, or partial service beats none.
What goes wrong
3 minOne deploy, config or expired certificate takes them all down.
Canary deploys, zones, and parallel maths as an upper bound only.
A 99.95% provider in series caps you at 99.95% or less.
Multiply the critical path before committing; design a fallback.
Teams prevent failures, then take 40 minutes to notice and roll back.
Cut MTTR: fast alerts, automatic rollback, one-command failover.
Trade-offs
3 min| Lever | Protects against | Costs |
|---|---|---|
| More redundancy | Hardware, server, zone failures | Idle capacity; not bad changes |
| Faster recovery | Every failure type, deploys included | Monitoring and automation work |
| Graceful degradation | Dependencies you don't control | A degraded mode to design and test |
- Karim · CTOLet's buy the next nine properly: a passive copy of StayHub in a second region. €9,000 a month.
- Lina · Senior engineer, your mentorAgainst a region going dark: that cost us zero minutes last year. Bad deploys cost us 368.
- Karim · CTORegion outages are rare but huge. A canary won't save us then.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Which availability StayHub promises, per journey, knowing the payment provider alone promises 99.95%.
- Where the money goes: ~70% of downtime was bad deploys, so faster rollback beats more servers.
- Whether the next nine is worth it: an hour of checkout outage costs ~€20,000; compare with what the nine costs every month.
I need an availability estimate for the checkout path before we answer a customer asking for 99.99%. Don't change code or infrastructure. Write docs/checkout-availability.md with each component and its availability (DNS 99.99%, load balancer 99.99%, app servers in 2 zones at 99.9% each, PostgreSQL with standby 99.95%, payment provider 99.95%), the series and parallel maths step by step, the end-to-end result, monthly downtime and the weakest link. Then use docs/incidents-2026.md to estimate cutting rollback from 40 minutes to 5. Mark every number measured, vendor-promised or assumed.
- Redo one step: the whole path comes to about 99.88%.
- No total is higher than its weakest component (that means something was added, not multiplied).
- A sentence on correlated failures (same release, same zone).
- It says "99.99% is achievable" by adding app servers.
- It treats copies on one release or in one zone as independent.
- It proposes multi-region active-active to fix downtime caused by bad deploys.
Check yourself
2 minNext episode: at 2:14 a.m. the database host in zone A dies, and the standby has to take over without anyone awake.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
| Availability | Per year | Per month | Per day |
|---|---|---|---|
| 99% (two nines) | 3.65 days | 7.3 h | 14.4 min |
| 99.5% | 1.83 days | 3.65 h | 7.2 min |
| 99.9% (three nines) | 8.76 h | 43.8 min | 1.44 min |
| 99.95% | 4.38 h | 21.9 min | 43.2 s |
| 99.99% (four nines) | 52.6 min | 4.38 min | 8.64 s |
| 99.999% (five nines) | 5.26 min | 26.3 s | 0.86 s |
You and Lina price the fourth nine
Karim needs a yes or no on 99.99% for checkout, and a cost. You have last year's incident list and the SLAs of everything on the checkout path.
- 1Set the budget in minutes
99.99% of a month = 4.4 minutes. Our current 99.9% = 43.8 minutes. Last year we were down about 526 minutes.
Why: Minutes tell you what kind of response is possible: 44 minutes allows a human; 4 minutes demands automation.
- 2Multiply the critical path
Series: DNS × LB × app tier × DB × payments.
tsconst series = (...a: number[]) => a.reduce((acc, x) => acc * x, 1); const parallel = (a: number, n: number) => 1 - Math.pow(1 - a, n); series(0.9999, 0.9999, parallel(0.999, 2), 0.9995, 0.9995); // 0.99880 → ~10.5 h/yearWhy: Your ceiling is set by the whole path, not your best component. 99.88% is already below 99.9% on paper.
- 3Find the weakest links
The app tier is ~99.9999%. The database and the payment provider are 99.95% each: together they're almost all of the paper downtime.
Why: Money spent on a box that's already strong buys almost nothing.
- 4Check what really broke
Last year: bad deploys ~368 min (70%), payment provider ~105 min (20%), everything else ~53 min. About one incident a month, 40 minutes each to notice and roll back.
Why: The formula assumes random hardware failures. Real downtime is mostly our own changes, which hit every copy at once.
- 5Pick the cheapest levers
1) Canary deploys + automatic rollback on burn rate: MTTR 40 → ~5 min, so 730 ÷ (730 + 0.083) ≈ 99.99% for deploy incidents. 2) A "pay at the hotel" fallback so checkout survives provider outages. 3) Make database failover automatic (today promoting the standby is a manual runbook), and measure it.
Why: Each lever attacks a real slice of the 526 minutes. A second region would attack the 10% "everything else" at the highest price.
- 6Answer with conditions
"Not today. After canaries and the payment fallback, we can offer 99.95% for checkout, and reconsider 99.99% after six months at that level."
Why: Sign only what the math and your track record support. A promise with a plan beats a promise with hope.
| Target | Typical architecture and practices | Rough cost multiple |
|---|---|---|
| 99% | One server, backups, restart during office hours | 1× |
| 99.9% | Redundant app servers behind a load balancer, managed DB with standby, on-call | ~2–3× |
| 99.99% | Multi-zone everything, automatic DB failover, zero-downtime deploys, canaries, 24/7 on-call | ~5–10× |
| 99.999% | Multi-region active-active, automated failover everywhere, chaos testing, strict change control | ~10–100× |
Interactive simulation
Visual diagram
One request path. Components in series multiply; redundant copies combine in parallel.
Analogy
- Catching a flightAlarm, train and shuttle must all workRunning a serviceParts in a chain multiply the risk
- Catching a flightEach is reliable, the trip less soRunning a serviceTotal is lower than each part
- Catching a flightTwo alarms: phone and clockRunning a serviceTwo copies of a server
- Catching a flightBoth alarms on one power socketRunning a serviceCopies sharing a zone or deploy
Where the analogy breaks: you catch a flight once. A service is used millions of times, so even tiny failure rates add up to real, angry users.
Check your understanding
Practice ladder
Does search meet its SLO on paper?
Estimation Warm-upStayHub's search SLO is 99.5% (from last episode). The search path:
- Managed DNS: 99.99%
- Load balancer: 99.99%
- 3 app servers, one per zone, each 99.9% (any one is enough)
- One search index cluster: 99.5%
Maya asks for a fourth app server "to make search more reliable". Trace the math: what's the path's availability, how many hours a month is that, does it meet the SLO, and would a fourth app server help?
Finish Lina's note on deploy incidents
Estimation PracticeLina's note for Karim has two gaps. Fill in steps 3 and 5.
- Budget: 99.99% = 4.4 min a month.
- Path: DNS 99.99% × LB 99.99% × app tier (2 zones) ~99.9999% × DB 99.95% × payments 99.95% ≈ 99.88%.
- Weakest links, and what they mean for 99.99%: ___
- Real incidents: about 1 a month, mostly bad deploys, 40 minutes to notice and roll back.
- Availability from incidents alone, at MTTR 40 min and at MTTR 5 min (MTBF 730 h): ___
The extranet's "99.9% uptime"
Estimation PracticeHotels update their rooms and prices in StayHub's extranet. Its pricing page says "99.9% uptime". Reality over the last year:
- A weekly 10-minute maintenance window (database restart) showing an error page.
- 2 incidents per quarter, each about 45 minutes.
What availability are hotels really getting? Does it meet the claim? Which single change would help most?
Which next nine is worth buying?
Decision StretchStayHub runs at about 99.9% (526 min of downtime last year). An hour of checkout outage costs about €20,000 (~100k bookings a day × ~€4.7 commission). Downtime came ~70% from bad deploys, ~20% from the payment provider, ~10% from everything else. No region outage in three years.
Karim can fund one project this year: - A. Passive copy in a second region: €9,000/month. - B. Canary deploys + automatic rollback: about €30,000 of engineering time once. Expected to cut deploy incidents from 40 to ~5 minutes. - C. "Pay at the hotel" fallback when the provider is down: about €20,000 once.
Write a short decision: "We chose X over Y because…, accepting the risk of…", with rough euro numbers.
Explain it yourself
A customer asks for five nines. Explain what that means, how you'd compute whether your system can achieve it, and how you'd respond.