Episode 71 · The average that lied

Latency percentiles and the tail

Tuesday, 10:00: the dashboard says search averages a healthy 120 ms. Maya has 214 support tickets saying search hangs, and session recordings show one search page in three taking over a second.

The average speed of your site describes nobody. Here's how to see what your unluckiest users actually live with.

30 min intermediatep50 / p95 / p99 latencyWhy average latency liesTail latency amplification with fan-outSLOs defined on latency percentilesRecording latency with histograms (HDR, aggregation)Taming the tail: hedging, timeouts, less fan-out

Problem
The hotel search page "feels slow" to many guests, yet the dashboard says average latency is a healthy 120 ms.
Why the simple system fails
The average blends fast and slow requests into a number nobody experiences; per-server percentiles get averaged into nonsense; and a page that calls 40 backends waits for the slowest one every time.
Engineering concept
Describe latency as a distribution: percentiles (p50, p95, p99, p99.9), recorded in mergeable histograms, with SLOs defined on the share of requests under a threshold, and designs that limit how the tail compounds.
Technology
Percentile math, histogram metrics (Prometheus buckets, HDR histograms), SLOs and error budgets, hedged requests, timeouts, fan-out reduction.
Trade-offs
Higher percentiles reveal more pain but need more data and are noisier; hedging cuts the tail at the price of extra load; finer histograms cost memory and storage.

The problem

2 min

Tuesday, 10:00. Last episode's fixes made search fast for a single guest.

Maya · Product manager
214 tickets this month: "search hangs". Recordings show about 1 search page in 3 takes over a second.
Omar · On-call SRE
Search API average: 120 ms. It's been green all month.
Lina · Senior engineer, your mentor
Both can be true. Guess how, before we look.

The big idea

4 min

Why one page in three is slow

The big idea

The average describes nobody. Judge latency by its slow end, measured where the guest waits, and remember that fan-out multiplies the tail.

The backend's 1-in-100 slow call becomes the guest's 1-in-3 slow page. So my first fix isn't a faster backend; it's fewer calls per page.

How it works

5 min

From one call to one page

1
Sort the calls
1,000 backend calls: 990 take ~108 ms, 10 take ~1,200 ms (a cold cache going to the database).
2
Read percentiles
A percentile like p99 = 400 ms means 99% of requests were faster. Here ≈ 105 ms, ≈ 1,200 ms. The slow end is the . The average, ~120 ms, is a number almost no call took.
3
Fan out
Fan-out: one page calling many backends at once. Search makes 40 calls in parallel, availability and price for 20 hotels, and shows nothing until all return.
4
Do the maths
Page slow if any call is slow: 1 − 0.99⁴⁰ ≈ 33%. With 2 calls: 1 − 0.99² ≈ 2%.
5
Merge, never average
Each pod counts durations in histogram buckets. Bucket counts add up across 30 pods; percentiles don't. Sum the buckets, then compute p99 once.
6
Set the goal on the page
An SLO: "99% of search pages under 400 ms over 28 days". Put a bucket boundary at exactly 400 ms so it's a count, not a guess.

Watch a decision being made

4 min

Worked example

You measure and shrink search's tail

Make the numbers show what guests feel, then bring slow pages toward the 1% the target allows. Lina checks the maths.

1Record histograms at the page
A page-level histogram covering all 40 calls, buckets with a boundary at 400 ms.
ts
```
const searchDuration = new Histogram({
  name: "search_duration_seconds",
  buckets: [0.05, 0.1, 0.2, 0.4, 0.8, 1.6, 3.2],
});
```
Why: Guests experience pages. Every backend can meet its own p99 while the page misses.
2Find the biggest lever with one line of maths
40 calls: ~33% slow pages. 2 calls: ~2%.
Why: Lina: "this tells you the biggest lever before you touch any code."
3Cut the fan-out, bound the rest
One availability call and one price call for all 20 hotels. A 300 ms timeout on reviews: show the page without them if late.
Why: Fewer chances to hit the tail, less load on the backends, and a page without reviews beats a page that hangs.
4Check the SLO
After a week: 98.3% of pages under 400 ms, up from about 67%. Still short of 99%: the batched availability call's own tail is what's left.
Why: "Now 'feels slow' is a number that can be met, missed, and owned."

In real life

3 min

Two well-known ways to deal with the tail

Amazon: targets deep in the tail

Its published Dynamo paper describes service agreements measured at the 99.9th percentile
Services deep in a page's fan-out are held to very strict targets
Each team owns its own tail

Fits many teams each owning a service in a deep call tree.

Google: cut the tail with extra requests

Its published "The Tail at Scale" article describes hedged requests
If a reply is late, send a copy to another replica and take the first answer
A little extra load; only safe for reads

Fits huge read fan-out where random slowness is unavoidable.

What goes wrong

3 min

Averaging percentiles across pods or time windows

The result isn't a percentile of anything; it hides problems.

Instead

Sum histogram buckets, then compute the percentile.

Alerting on average latency

A regression hitting 3% of requests barely moves the mean.

Instead

Alert on p95/p99, or on SLO error-budget burn.

Load-testing with users who wait for each reply

Fewer requests are sent exactly when the system stalls, so the tail looks great ().

Instead

Fixed-rate load tests, measured from the intended send time.

Trade-offs

3 min

Tail fix	Gains	Costs	Choose when
Reduce fan-out	Fewer chances to be slow, less load	API changes	Pages with tens of calls
Timeouts + partial results	Hard cap on page time	Some pages incomplete	Non-critical parts (reviews)
Hedged requests	Big cut for random slowness	A few % extra load; reads only	Spare capacity, uncorrelated slow replicas

Thursday, 14:00. After batching, the availability call still has a p99 of 1.1 s.

Omar · On-call SRE
Hedge it: no answer in 150 ms, send a copy to another pod. Page p99 drops to about 350 ms for ~4% more traffic.
Karim · CTO
Availability runs at 70% CPU on Sunday evenings.
Karim · CTO
If the slowness is overload, hedging sends more traffic exactly when it's drowning.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which number defines "fast": the SLO, 99% of search pages under 400 ms over 28 days, not the 120 ms average.
Where latency is measured: at the page, where all parallel calls add up, not only per service.
Which tail fixes are worth their cost: batching first; hedging only for idempotent reads with spare capacity.

Tell Claude Code

Search feels slow for many guests though the average is 120 ms. Record page latency at the search aggregator into a histogram with a bucket boundary at 400 ms. Build service-wide p50, p95 and p99 by summing bucket counts across all pods, never averaging per-pod percentiles. Alert on the SLO, 99% of pages under 400 ms over 28 days, with a burn-rate alert. Don't change search code yet. Show me the plan, the query behind each panel, and why each can't hide 3 bad pods out of 30.

Check before you accept

The query sums buckets across pods, then computes the percentile; no avg() of per-pod p99s.
A bucket boundary sits exactly at 400 ms.
Latency is measured where the guest waits (the page).

Red flags

It reports or alerts on the average latency.
It averages percentiles across pods, regions or time windows.
It adds hedging to non-idempotent calls, or to an overloaded service.

Check yourself

2 min

Q1/3Reasoning

100 requests: 95 take 40 ms and 5 take 3,000 ms. Which statement is true?

How confident are you?

Next episode: after a release, booking-api's p95 jumps from 250 ms to 2 s, and Karim proposes rewriting it in Go.

Finished?

Mark it complete to update your skill map.

Next: Profiling: measure before you optimise

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Measuring the tail correctly

You fix how search is measured and how its tail behaves. Lina checks the maths at two points.

Worked example

You measure and shrink search's tail

Search averages 120 ms, but ~1 page in 3 takes over a second. Make the numbers show what guests feel, then bring slow pages down toward the 1% the target allows.

1Record histograms, not averages
Each search pod counts request durations in fixed buckets, with a boundary at 400 ms (the target).
ts
```
const searchDuration = new Histogram({
  name: "search_duration_seconds",
  buckets: [0.05, 0.1, 0.2, 0.4, 0.8, 1.6, 3.2],
});
```
Why: Bucket counts can be added across pods and time windows; percentiles can't. A boundary at the target makes "share under 400 ms" exact, not interpolated.
2Measure the page, not just the calls
Add a page-level histogram at the search endpoint (or the load balancer) that covers all 40 backend calls.
Why: Guests experience pages. Every backend can meet its own p99 while the page misses its target.
3Compute the fan-out effect
Page slow if any call is slow: 1 − 0.99⁴⁰ ≈ 33%. With 2 calls: 1 − 0.99² ≈ 2%.
Why: Lina: "this one line of maths tells you the biggest lever before you touch any code."
4Cut the fan-out
One availability call and one price call, each for all 20 hotels, instead of 40 separate calls.
Why: Fewer chances to hit the tail, and less total load on the backends.
5Bound what's left
Timeout of 300 ms on reviews (not critical): show the page without them if late. Keep backend CPU below ~70% so requests don't queue.
Why: A page without reviews beats a page that hangs. Queueing at high utilisation is the most common source of tail latency.
6Set the SLO and check it
"99% of search pages under 400 ms over 28 days." After a week: 98.3%, up from about 67%, still short of 99%: the batched availability call's own tail is what's left (the trade-off below).
Why: Lina: "now 'feels slow' is a number that can be met, missed, and owned."

Statistic	What it tells you	Use it for	Watch out
Mean	Total time / requests	Capacity maths (total work), cost	Hides the tail; dominated by outliers
p50	The typical request	Spotting broad regressions	Says nothing about slow users
p95 / p99	The slow but common experience	SLOs, alerts, user-facing targets	Needs enough samples; noisy at low traffic
p99.9 / max	Extreme outliers	Fan-out services, debugging	Very noisy; max is a single request

Report several percentiles side by side; one number is never enough.

Interactive simulation

Visual diagram

Guest

Search aggregator

Availability (slow 1%: ~1.2 s)

Pricing (slow 1%: ~1.2 s)

Reviews (p99 120 ms)

Photos (p99 90 ms)

Histograms + SLO

Click a component

One search page fans out to several backends. The page is as slow as its slowest call.

Analogy

Hotel check-inRequest latency

Hotel check-in"Average check-in: 4 minutes"Request latencyThe mean, describing nobody
Hotel check-inA typical guest: 3 minutesRequest latencyp50
Hotel check-inArriving with the coach party: 25 minutesRequest latencyp99, who writes the review
Hotel check-inA group of 40 waits for the last guestRequest latencyFan-out: the slowest call wins
Hotel check-inAveraging three hotels' tallest guestsRequest latencyAveraging percentiles: meaningless

Where the analogy breaks: a hotel sees a few hundred guests a day. A busy service sees millions of requests, so even the p99.9 is thousands of real, unhappy waits.

Check your understanding

Q1/4Decision

Match each goal to the most suitable measure:

Promise to users that search feels fast

Estimate how many CPU-seconds the service needs

Detect a regression affecting 2% of checkouts

Set a target for a backend called 50× per page

How confident are you?

Practice ladder

Step 1 · Trace

Read the distribution

Estimation Warm-up

The hotel-page API handled 1,000 requests in one minute:

900 took 60 ms
80 took 150 ms
15 took 900 ms
5 took 4,000 ms

Compute the average, p50, p95 and p99 (nearest rank), and say which number you'd show Maya and why.

Step 2 · Complete

Finish the SLO computation

Estimation Practice

Search runs on 3 pods. Over 28 days, their histogram counts for the bucket boundary at 400 ms:

Pod	Requests ≤ 400 ms	Total requests
A	4,950,000	5,000,000
B	4,970,000	5,000,000
C	1,850,000	2,000,000

The SLO is 99% of search requests under 400 ms over 28 days.

Compute the service-wide share under 400 ms: ___
Is the SLO met? ___
A teammate averages the three pods' shares instead. What does that give, and why is it wrong?

Step 3 · Solve

The owner dashboard's tail

Design Practice

After episode 1's fix, the owner dashboard makes 12 backend calls in parallel: bookings, reviews, revenue, messages, payouts, and occupancy for each of the next 7 days (7 calls). Each call is slow (> 800 ms) about 2% of the time, independently. Everything else is fast.

What share of dashboard loads hit at least one slow call?
Propose two changes, and compute the new share for the one that changes the number of calls.
Which call would you put a timeout on with a partial result, and which never?

Step 4 · Decide

Which percentile for checkout's SLO?

Decision Stretch

Karim wants a latency SLO for checkout. Facts:

Checkout handles about 40 checkouts a minute on average (~57,000 a day), with peaks around 100 a minute on Friday evenings (the ~180 checkout API requests/s from episode 1 are all the calls those booking forms make).
Omar proposes p99.9 < 1 s, "because checkout is money".
Maya proposes p95 < 600 ms, "because that's what most guests feel".
Alerts must not page people for noise.

Write a short decision: "We chose X over Y because…, accepting the risk of…". Include the window.

Explain it yourself

Your manager says: "Average latency is 120 ms, so search is fast." Explain why that might be wrong and what you'd measure instead.

0 words