Episode 79 · Was last month a good month?

SLIs, SLOs, SLAs and error budgets

Monday, 10:04: Omar's dashboard says StayHub was up 100% last month, yet 12,000 checkouts failed on Saturday alone. A hotel chain wants a 99.99% promise in its contract by Friday, and nobody can say if last month was good.

Three confusing acronyms, one simple idea: measure what users feel, set a goal, and promise a bit less than you deliver.

30 min intermediateSLI vs SLO vs SLAError budgetsBurn-rate alertingAvailability nines and downtime budgetsp50 / p95 / p99 latencyChoosing an availability target

Problem
Product wants “high availability”, sales wants to promise 99.99%, engineering wants to ship faster, and nobody agrees whether last week was a good week.
Why the simple system fails
Without shared numbers, reliability is argued by feelings: alerts fire on CPU spikes nobody cares about, real user pain goes unnoticed, and every outage triggers a freeze on releases.
Engineering concept
Measure what users experience (SLI), set an internal target (SLO), promise a looser one externally (SLA), and treat 1 − SLO as an error budget you can spend.
Technology
Request success and latency metrics at the edge, SLO recording rules, multi-window burn-rate alerts, SLO dashboards, contractual service credits.
Trade-offs
Tighter SLOs cost more infrastructure, on-call effort and slower releases. Looser ones risk losing users. The right number depends on what users and the business actually need.

The problem

2 min

Monday, 10:04. StayHub now serves about 40 million visits a month.

Omar · On-call SRE
Uptime last month: 100%. The homepage ping never failed once.
Maya · Product manager
Then explain the 300 emails from guests who couldn't pay on Saturday.
Karim · CTO
And a 120-hotel chain wants 99.99% in their contract by Friday. Can we sign?

The big idea

4 min

Green dashboard, failed bookings

The big idea

Measure what guests feel. Aim a little below what you achieve. Promise customers a little less than your aim.

So last month was acceptable but close: we met the goal with 30% of the budget left, and one Saturday afternoon ate most of the month. The ping saw none of it.

How it works

5 min

From one request to a decision

1
Count at the edge
The load balancer sees every checkout request, even ones that never reached a healthy server.
2
Score each one
Good: no 5xx and under 2 s. Valid: everything except the guest's own errors (4xx).
3
The SLI
An (indicator) measures what users feel: good ÷ valid, over a rolling 30 days. Last month: 99.93%.
4
The SLO
An (objective) is the team's goal for it: 99.9% over 30 days, so 20,000 of 20 million may fail. We had 14,000.
5
The error budget
The is 1 − SLO: the failures allowed, to spend on releases and bad luck. 70% spent. Budget left: keep shipping. Empty: reliability first.
6
The SLA
An (agreement) is a promise to customers with money attached. Always looser than the SLO: SLA ≤ SLO ≤ what we achieve. 99.99% is 4.3 minutes a month.

Watch a decision being made

4 min

Worked example

Lina writes the checkout SLO

Turn "checkout should be reliable" into a number StayHub can measure, steer by and alert on. About 20 million checkout requests a month.

1Define good and valid, for one journey
Checkout first: where guests pay. Valid = all except 4xx. Good = valid, not 5xx, and under 2 s.
Why: One SLO per journey guests care about. A 9-second request is a failure even if it returns 200.
2Set the target from data
Last month: 99.93%. SLO: 99.9% over a rolling 30 days, just below.
Why: A target above reality is a permanent emergency; one far below hides decline.
3Turn it into a budget
(1 − 0.999) × 20,000,000 = 20,000 bad requests, about 43 minutes of full outage.
Why: "20,000 failed checkouts, 14,000 used" is something product and engineering can plan around.
4Alert on burn rate
Page when the 1-hour and 5-minute error ratios exceed 14.4 × 0.1% = 1.44% (2% of the monthly budget in an hour). Ticket when the 3-day ratio exceeds 0.1%.
yaml
```
- alert: CheckoutBudgetFastBurn
  expr: slo:checkout_error_ratio:rate1h > (14.4 * 0.001)
    and slo:checkout_error_ratio:rate5m > (14.4 * 0.001)
  labels: { severity: page }
```
Why: The long window ignores blips; the short one clears the alert minutes after the fix.

In real life

3 min

Two well-known approaches

Google: internal SLOs steer releases

Its public SRE book describes SLOs with an error budget agreed with product teams
When a budget runs out, risky launches pause until reliability recovers
It even describes planned outages so teams wouldn't depend on more than the SLO

The number decides how fast the team may move.

AWS: public SLAs with service credits

Publishes an SLA per service with monthly uptime targets
Below target, customers claim a credit on their bill, not full damages
Customers design their own redundancy on top

The number tells customers what they can count on.

What goes wrong

3 min

Server metrics (CPU, memory) as SLIs

CPU can be at 20% while every checkout fails on a dependency.

Instead

Success and latency of real requests, close to guests.

SLA equal to the SLO, or aiming for 100%

Every miss costs money; 100% forbids change and is unreachable.

Instead

SLA looser than SLO, SLO just below current performance.

An error budget with no policy

If nothing changes when it's empty, the SLO is just a dashboard.

Instead

Agree with product in advance: pause risky launches, reliability first.

Trade-offs

3 min

SLO	Allowed per month	Needs	Fits
99.99%	~4 minutes	Automatic failover everywhere, slow releases	Payments infrastructure, platforms
99.9%	~43 minutes	Redundancy in one region; a human can roll back	Most consumer products, booking
99%–99.5%	Hours	Simple infrastructure	Internal tools, prototypes

Wednesday, 16:00. The hotel chain's contract is on the table.

Karim · CTO
The chain is worth €400k a year. Competitors promise 99.99%. Let's sign and make it true.
Lina · Senior engineer, your mentor
That's 4.3 minutes a month. One bad deploy takes 20 minutes to roll back. We'd owe credits most months.
Karim · CTO
Then we invest: automatic failover, slower releases. That's the price of big customers.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What counts as a good checkout: non-5xx and under 2 s, at the load balancer. Left to Claude, it picks what's easy to measure.
The SLO, 99.9% over 30 days (20,000 bad checkouts), and the budget policy when it's empty, agreed with product before an incident.
Whether to sign a 99.99% SLA: tighter than our own SLO and our payment provider.

Tell Claude Code

Add a checkout SLI and burn-rate alerts. Plan first. Valid events: POST /checkout requests at the load balancer, excluding 4xx; good events: non-5xx responses under 2 seconds. SLO 99.9% over a rolling 30 days (~20,000 bad requests). Use our existing load balancer metrics; no new monitoring vendor. Page when the burn rate is above 14.4 over 1 hour and still above it over the last 5 minutes; ticket for slow burn over 3 days. Show me the queries and what last month's SLI and remaining budget would have been.

Check before you accept

Last month's SLI with the new query lands near 99.93% (~14,000 bad). If it says 100%, it measures the wrong thing.
The query counts slow requests (over 2 s) as bad and excludes 4xx.
At 3 a.m. with almost no traffic, one failed request doesn't page anyone.

Red flags

It uses average latency or CPU as the SLI.
It measures inside the app only, so a total outage shows zero errors.
It "fixes" a noisy alert by raising the threshold until it never fires.

Check yourself

2 min

Q1/3Reasoning

Your checkout SLO is 99.9% over 30 days. You expect 20 million valid checkout requests in that window. How many failed requests does the error budget allow?

requests

How confident are you?

Next episode: Karim asks what "four nines" would really take, and you count every box a checkout passes through.

Finished?

Mark it complete to update your skill map.

Next: The nines: how much availability do you need?

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Paging on budget burn, not on blips

A useful SLI states the event, what counts as good, what counts as valid, where it's measured and the window. Most fall into a few families:

SLI type	Good event	Typical for
Availability	Request completed without a server error (non-5xx)	APIs, web pages
Latency	Request completed under a threshold (e.g. 2 s)	User-facing endpoints
Freshness	Data served is younger than X (e.g. rooms left < 60 s old)	Caches, search indexes
Correctness	Output matches expected (price equals recomputed price)	Billing, pricing
Completion	Job finished within its deadline	Confirmation emails, payouts

Pick 1–3 SLIs per critical guest journey, not one per server.

Worked example

Lina writes the checkout SLO

Turn "checkout should be reliable" into a number StayHub can measure, steer by and alert on. About 20 million checkout requests a month go through the load balancer.

1Pick the journey
Start with checkout: it's where guests pay, and where Saturday's failure hit. Search comes next, with its own SLO.
Why: One SLO per journey guests care about. An SLO per server measures our machines, not their experience.
2Define good and valid
Valid = every checkout request except 4xx (the guest typed a bad card number). Good = valid and not 5xx and under 2 s.
Why: Counting guest mistakes as failures would punish us for things we can't fix; a request that takes 9 s is a failure even if it returns 200.

3Measure where guests are

Use the load balancer's logs, plus a synthetic probe that runs a test booking every minute for quiet night hours.

yaml

- record: slo:checkout_error_ratio:rate1h
  expr: sum(rate(http_requests_total{route="/checkout", code=~"5.."}[1h]))
      / sum(rate(http_requests_total{route="/checkout", code!~"4.."}[1h]))
# same pattern for 5m, 6h and 3d windows; slow requests (> 2 s) also count as bad

Why: The edge sees requests that never reached a server. The probe keeps the number meaningful at 3 a.m. when few guests book.

4Set the target from data
Last month we achieved 99.93%. Set the SLO at 99.9% over a rolling 30 days, just below what we do today.
Why: A target above reality is a permanent emergency; one far below it hides decline. Slightly below current performance is honest and reachable.
5Turn it into a budget
Budget = (1 − 0.999) × 20,000,000 = 20,000 bad requests, or about 43 minutes of full outage.
Why: "99.9%" is abstract. "20,000 failed checkouts, 14,000 already used" is something product and engineering can plan around.

6Alert on burn rate

Page when the 1-hour and the 5-minute error ratio are above 14.4 × 0.1% = 1.44%. Open a ticket when the 3-day ratio is above 0.1%.

yaml

- alert: CheckoutBudgetFastBurn   # 2% of the monthly budget in 1 hour
  expr: slo:checkout_error_ratio:rate1h > (14.4 * 0.001)
    and slo:checkout_error_ratio:rate5m > (14.4 * 0.001)
  labels: { severity: page }
- alert: CheckoutBudgetSlowBurn   # 10% of the monthly budget in 3 days
  expr: slo:checkout_error_ratio:rate3d > 0.001
  labels: { severity: ticket }

Why: The long window ignores blips; the short one makes the alert stop as soon as the problem is fixed. Slow burns matter but can wait for daylight.

Interactive simulation

Visual diagram

Guests

Synthetic probe

Load balancer

Booking API

Database

SLO engine

Burn-rate alerts

Click a component

Where SLIs are measured and how they flow into SLOs, alerts and SLA reports.

Analogy

A train companyYour service

A train companyShare of trains on time, checked at the platformYour serviceSLI: share of good requests
A train companyInternal goal: 97% on timeYour serviceSLO: the team's target
A train companyRefund for season tickets below 92%Your serviceSLA: the contract with customers
A train company3% of trains may be lateYour serviceError budget: allowed failures
A train companyStorm used the budget? Postpone track worksYour serviceBudget gone? Pause risky releases

Where the analogy breaks: train delays come in big, visible chunks. Software failures are often tiny and invisible, like one slow request in a thousand, so you only see them if you measure.

Check your understanding

Q1/3Recall

Match each statement to the term:

99.2% of checkout requests succeeded in under 2 s last week

99.9% of checkout requests should succeed in under 2 s over 30 days

Below 99.5% monthly availability, customers receive a 10% credit

0.1% of requests may fail this month

How confident are you?

Practice ladder

Step 1 · Trace

Score last week's search requests

Estimation Warm-up

StayHub's search SLO: 99.5% of valid requests are good over 30 days. Good = not 5xx and under 1 s. Valid = everything except 4xx and the load balancer's own health checks.

Last week's load balancer log summary:

Requests	Count
Total	10,500,000
Health checks (`GET /healthz`)	500,000
4xx (bad input)	200,000
5xx	18,000
2xx but slower than 1 s	22,000

Trace the numbers: how many valid requests, how many bad, what's the SLI, and does last week alone stay within the SLO?

Step 2 · Complete

Finish Lina's alert plan

Design Practice

Checkout SLO: 99.9% over 30 days (budget 0.1%). Lina's alert plan has two gaps. Fill in 2 and 4.

Fast burn, page: 1-hour error ratio > 14.4 × 0.1% = 1.44% and 5-minute ratio > 1.44%.
Medium burn, page: 6-hour error ratio > ___ (burn rate 6), with a short confirming window of ___.
Slow burn, ticket: 3-day error ratio > 0.1% (burn rate 1).
What a slow-burn ticket means in budget terms: ___

Step 3 · Solve

An SLO for confirmation emails

Design Practice

After paying, guests get a confirmation email sent by a background worker. Maya: "Guests panic and call support if the email takes more than 5 minutes." About 3 million confirmation emails go out each month (one per booking). Last month, 99.7% arrived within 5 minutes; a queue backlog once delayed 4,000 emails by 40 minutes.

Write the SLI and SLO for this journey, compute the budget, and say how you'd alert on it.

Step 4 · Decide

Release freeze or launch?

Decision Stretch

Day 18 of the 30-day window. The checkout budget is 95% spent after two incidents (a bad deploy and a payment provider outage). Maya wants to launch the redesigned checkout page tomorrow: it's promised to a marketing campaign starting next week.

StayHub's written policy says: "When the budget is exhausted, only reliability fixes and urgent patches ship."

Write a short decision in this form: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain SLIs, SLOs, SLAs and error budgets to a product manager, and why they make release decisions easier.

0 words