Episode 74 · Five teams, one second too many

Latency budgets and the critical path

Monday, 9:00: booking confirmation takes 1.3 seconds at p95. Maya wants 400 ms before the winter campaign, and five different teams each own a piece of the request.

"Make it fast" isn't a plan. A latency budget turns it into a number for each step, so you know exactly where to look.

30 min intermediateLatency budgets across hopsShortening the critical path (parallel and async work)Payload size and compressionLatencyp50 / p95 / p99 latencyTimeouts & deadline budgets

Problem
The hotel booking confirmation takes 1.4 s. Product wants it under 400 ms at p95, and five teams each own a piece of the request.
Why the simple system fails
Without a budget, every team optimises its own piece "a bit", nobody owns the total, and new features quietly add 50 ms each until the page is slow again. Calls made one after another add up even when each is fast.
Engineering concept
Set a total target, split it into allowances per hop, and attack the critical path: run independent work in parallel, move work the user doesn't need to wait for off the request, and shrink round trips and payloads.
Technology
Distributed traces to see the critical path, Promise.all / concurrent calls, queues and workers for async work, CDN and connection reuse, gzip/Brotli compression, per-hop timeouts and deadlines.
Trade-offs
Parallel calls add load and make failures partial; async work makes results eventually consistent; compression costs CPU; caching costs freshness. Spend complexity only where the budget is blown.

The problem

2 min

Monday, 9:00. Three weeks before the winter campaign.

Maya · Product manager
Booking confirmation: p95 1.3 s. Guests double-click and some book twice. I need 400 ms before the campaign.
Karim · CTO
Inventory, risk, finance, payments and notifications each own a piece. Five teams. Who fixes what?
Lina · Senior engineer, your mentor
You lead this one. I'll only check your answers.

The big idea

4 min

Only the critical path counts

The big idea

Only the critical path counts. Shorten it: remove, parallelise, defer, then make cheaper. Then give every step an allowance and a timeout.

The biggest slices weren't slow code: they were the email and PDF, 580 ms the guest should never have waited for.

How it works

5 min

Shortening the critical path

1
Draw the path
The critical path is the steps that must happen one after another before the response. Today all ten are: network 60 → auth 20 → availability 100 → fraud 150 → tax 120 → payment 180 → save 40 → email 300 → PDF 280 (+30 logic) = 1,280 ms.
2
Defer what can wait
Async work goes to a queue for a background worker. The guest needs "confirmed", not the email or PDF: enqueue one job (~5 ms). → ~705 ms.
3
Parallelise the independent
Availability, fraud and tax don't need each other: run them together. Parallel costs the slowest: 370 → 150 ms. → ~485 ms.
4
Make a step cheaper
Payment opened a new TLS connection per call (~80 ms). Reuse a kept-alive one. → ~405 ms.
5
Write the budget
A splits the target into allowances with owners: platform 80, checks 150, payment 100, bookings 70 = 400 ms. Each allowance becomes a timeout: the longest you wait before a fallback.
6
Verify end to end
Real traces: p95 ≈ 390 ms. Percentiles of parts don't add up, so only the end-to-end number counts.

Watch a decision being made

4 min

Worked example

You bring confirmation from 1.3 s to 400 ms

Reach 400 ms p95 across 10 steps owned by five teams, without making a booking less safe. Lina checks twice.

1Map the dependencies
Payment needs tax (the total); save needs payment; email and PDF need save. Availability, fraud and tax need nothing from each other.
Why: Dependencies decide what can move. Without the map, you'd ask the wrong team to speed up.
2Move side effects off the path
After saving, enqueue booking.confirmed (~5 ms). A worker sends the email and builds the PDF, with retries.
Why: The worker can retry a failed email; a guest's request can't.
3Run the checks together, each with a timeout
Promise.all over the three checks, each limited by the 150 ms allowance.
ts
```
const [avail, risk, tax] = await Promise.all([
  inventory.recheck(req, { timeoutMs: t(150) }),
  fraud.score(req, { timeoutMs: t(150) }),
  taxes.quote(req, { timeoutMs: t(150) }),
]);
```
Why: Lina: "fraud is now the slowest, so it's the only one worth optimising. And a budget without timeouts is a wish."
4Decide the fallbacks, then verify
Tax times out: "we couldn't confirm the price, please retry" instead of hanging. Fraud times out: a stricter rule set. Then end-to-end p95 on real traces: ~390 ms.
Why: Each allowance needs a plan for when it's blown, or one slow provider holds every worker.

In real life

3 min

What must happen before "done"?

Amazon: "order placed" first, the rest later

As customers see it, the order is accepted at once; emails arrive afterwards
Some checks finish later; an order can occasionally be cancelled
Very fast checkout; failures handled after the customer left

Fits huge volumes where a few later cancellations are acceptable.

Ticket sellers: confirm everything first

As customers see it, sites like Ticketmaster hold a "processing" screen until seat and payment are confirmed
A seat can't be resold, so the answer must be final
Slower, but "confirmed" really means confirmed

Fits scarce inventory where a later cancellation is a broken promise.

What goes wrong

3 min

No timeouts on dependencies

One slow provider holds shared workers and connections until unrelated pages fail.

Instead

A timeout from the budget, a fallback, and a .

Optimising a step off the critical path

A faster async email or a faster non-slowest parallel call changes nothing the guest feels.

Instead

Optimise the slowest critical step, then re-check which one that is.

Making everything async

Saying "confirmed" before checking availability means cancelling real trips later.

Instead

Keep on the path whatever decides the answer; defer only side effects.

Trade-offs

3 min

Option	Gains	Costs	Use for
Synchronous	The answer is final	Every step adds latency	Availability, payment, save
Async (queue)	Off the path, retries	Failures after the guest left	Email, PDF, analytics
Parallel	Sum becomes max	More concurrent load, partial failures	Independent checks

Wednesday, 16:00. The campaign doubles traffic; Karim wants margin.

Karim · CTO
Take payment off the path too. Say "confirmed", authorise the card a few seconds later. 100 ms of margin.
Maya · Product manager
Then about 2% of guests get "confirmed" followed by "your payment failed". Worse than waiting 100 ms.
Karim · CTO
We'd hold the room 15 minutes and email them to retry. Most will.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The target and percentile: confirmation under 400 ms at p95, down from 1,280.
What the guest waits for: availability, fraud, tax, payment and saving stay; email (300 ms) and PDF (280 ms) move to a queue.
Each owner's allowance, the timeout it becomes, and what the guest sees when a call times out.

Tell Claude Code

Bring POST /bookings/confirm from 1,280 ms p95 to under 400 ms without new infrastructure, one change per commit: move the confirmation email and PDF to our job queue; run the availability re-check, fraud check and tax quote in parallel since none depends on the others; keep a warm connection to the payment provider. Give every outside call a timeout from its allowance (checks 150 ms) and tell me what the guest sees on a timeout. Don't change payment or booking logic. Plan first, then measure end-to-end p95 after each step in staging.

Check before you accept

p95 after each step: about 705, then 485, then 405 ms. A step that doesn't move it wasn't on the critical path.
Only truly independent calls run in parallel; payment still waits for the checks.
Every outside call has a timeout; the queued email and PDF have retries and an alert.

Red flags

It optimises a 20 ms piece while 580 ms of email and PDF sit in line.
It moves something that decides the response (room held, payment authorised) to the background.
It adds per-hop p95s and calls the sum the end-to-end p95.

Check yourself

2 min

Q1/3Reasoning

A hotel page needs three independent calls: availability 120 ms, prices 80 ms, reviews 60 ms. Today they run one after another. If you run them in parallel, how long does that section take (ignoring overhead)?

How confident are you?

Next module: StayHub's first TV ad airs on Sunday, traffic may triple in 30 seconds, and Karim asks: bigger servers, or more of them?

Finished?

Mark it complete to update your skill map.

Next: Scaling up vs scaling out

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Spending a 400 ms budget

You lead the whole change. Lina's two checks are in the "why" lines.

Worked example

You bring confirmation from 1.3 s to 400 ms

Booking confirmation is 1,280 ms at p95 across 10 sequential steps owned by five teams. Reach 400 ms p95 without making a booking less safe.

1Map the critical path
From traces, list each step, its p95 and what it depends on. Payment depends on tax (the total); save depends on payment; email and PDF depend on save.
Why: Dependencies decide what can move. Without the map, you'd ask the wrong team to speed up.
2Move side effects off the path
After saving the booking, enqueue booking.confirmed (~5 ms). A worker sends the email and builds the PDF, with retries.
Why: 580 ms the guest doesn't need to wait for. The worker can retry a failed email; a guest's request can't.
3Run independent steps together
Promise.all over availability, fraud and tax: the time becomes the slowest of the three, 150 ms instead of 370.
ts
```
const [avail, risk, tax] = await Promise.all([
  inventory.recheck(req, { timeoutMs: t(150) }),
  fraud.score(req, { timeoutMs: t(150) }),
  taxes.quote(req, { timeoutMs: t(150) }),
]);
```
Why: Lina: "now fraud is the slowest, so it's the only one of the three worth optimising. Speeding up tax changes nothing."
4Make the remaining steps cheaper
The payment client opened a new TLS connection per call. Keep connections alive: 180 → 100 ms.
Why: A handshake is pure overhead on the critical path. Connection reuse is free speed.
5Give every call a timeout from the budget
Each call gets min(its allowance, time left). If tax times out, show "we couldn't confirm the price, please retry" instead of hanging; if fraud times out, fall back to a stricter rule set.
Why: Lina: "a budget without timeouts is a wish. One slow provider would hold every worker."
6Write the budget and verify
Allowances: platform 80, checks 150, payment 100, bookings 70 = 400 ms, with owners. End-to-end p95 on real traces: ~390 ms.
Why: Percentiles of parts don't add up exactly; only the end-to-end measurement says whether the target is met.

Lever	What it saves	Cost / risk
Parallelise independent calls	Sum → max of the calls	More concurrent load on dependencies; partial failures to handle
Async (queue + worker)	The whole step leaves the critical path	Eventual consistency; failures happen after the user left (retries, alerts)
Cache / precompute	The step costs ~1 ms or nothing	Staleness and invalidation
Connection reuse, CDN, closer region	Handshake and distance round trips	Infra cost; multi-region data complexity
Compression, smaller payloads	Transfer time, especially on mobile	A little CPU to compress
Faster code or query	Time inside one hop	Engineering time; only matters on the critical path

Interactive simulation

Visual diagram

Guest browser

CDN edge

Booking API

Availability re-check

Fraud check

Tax quote

Payment API

Job queue

Worker (email, PDF invoice)

Click a component

StayHub's booking confirmation (POST /bookings/confirm). Walk through the budget, the critical path and the levers.

Analogy

Cooking dinner for 19:00A latency budget

Cooking dinner for 19:00Guests arrive at 19:00A latency budgetThe total time target
Cooking dinner for 19:00Chop, then rice, then sauce: 45 minutesA latency budgetCalls one after another
Cooking dinner for 19:00Chop while the rice cooks: 25 minutesA latency budgetCalls side by side
Cooking dinner for 19:00Washing up after dinner is servedA latency budgetWork done after responding
Cooking dinner for 19:00Sauce made yesterday, just reheatA latency budgetA cache

A faster knife saves nothing if chopping already happens while the rice cooks. Where the analogy breaks: dinner happens once. A request runs millions of times, and its slow 5% is what you budget for.

Check your understanding

Q1/3Decision

Which steps of a booking request can safely move off the critical path into a background job? (Select all that apply.)

Select all that apply.

How confident are you?

Practice ladder

Step 1 · Trace

Trace the new confirmation

Estimation Warm-up

The new confirmation path (p95 per step): network 60, auth 20, then in parallel availability 100 / fraud 150 / tax 120, then payment 100, save 40, enqueue 5, logic 25.

Compute the critical path for each scenario: 1. Normal day. 2. Fraud has a slow evening: 400 ms (its timeout is 150 ms, after which a stricter rule set is applied in 10 ms). 3. The email worker is backed up and emails take 2 minutes to send.

Step 2 · Complete

Finish the budget table

Design Practice

The hotel page has a p95 budget of 300 ms. Fill in the missing allowances and say which steps run in parallel.

Step	Depends on	Measured p95	Allowance	Owner
Network + TLS	–	50	50	platform
Hotel details (cached)	–	5	___	catalog
Prices for the dates	–	60	___	pricing
Availability	–	80	___	inventory
Reviews summary	–	120	___	reviews
Render	all above	40	40	web

Total must be ≤ 300 ms. Reviews are nice to have.

Step 3 · Solve

Speed up the owner's payout page

Design Practice

The owner payout page takes 2.1 s. Its trace, all sequential:

auth                          20 ms
load owner profile            30 ms
load hotels (12)              40 ms
for each hotel: payouts      ~90 ms x 12 = 1,080 ms
exchange rates API           310 ms   (rates change daily)
build CSV export             450 ms   (only used when the owner clicks "Download")
render                       170 ms

Target: 500 ms p95. Propose changes, in order, with the expected time after each.

Step 4 · Decide

A new feature wants 60 ms

Decision Stretch

Confirmation is at ~390 ms against a 400 ms p95 budget. The marketing team wants to show "Guests who booked this also liked…" recommendations on the confirmation page, computed during the request: about 60 ms p95.

Options: - A: add it to the critical path and raise the budget to 460 ms. - B: load recommendations from the browser after the confirmation is shown, with their own 300 ms budget and hidden if late. - C: don't build it.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain how you would take a 1.3 s booking confirmation to under 400 ms without buying new infrastructure.

0 words