Timeouts, backoff and circuit breakers
Friday, 21:10: a deploy makes the reviews service answer in 8 s instead of 80 ms. By 21:14, every StayHub page, booking included, is timing out, although only reviews are sick.
A broken service is annoying. A slow one can freeze your whole platform. Here's the small toolkit that stops that from happening.
- Problem
The pricing service goes from 50 ms to 8 s. Ten minutes later, search, booking and checkout are all down.
- Why the simple system fails
No timeouts means threads pile up waiting; naive retries multiply traffic onto the sick service; shared pools let one dependency starve everything else.
- Engineering concept
Bound every wait, retry rarely and with randomised backoff, stop calling what is clearly broken, isolate resources, and reject early when overloaded.
- Technology
Client timeouts and deadline propagation, exponential backoff with full jitter, retry budgets, circuit breakers (closed/open/half-open), bulkhead pools, bounded queues and 429/503 load shedding.
- Trade-offs
Every knob trades a little availability or latency in the normal case for survival in the bad case, and badly tuned values cause their own outages.
The problem
2 min- Omar · On-call SREReviews went from 80 ms to 8 s at 21:10. Now every page times out. Search, login, booking.
- Karim · CTOBooking doesn't even use reviews. How is it down?
- Lina · Senior engineer, your mentorIt shares workers with something that does. Guess first.
The big idea
4 minA slow dependency is worse than a dead one. Bound every wait, and give every optional feature a way to step aside.
Adding workers doesn't fix it: at 8 s we'd need about 1,200, and more if reviews got slower. What we were short of was limits. With them, Friday would have been "hotel pages without stars", not an outage.
How it works
5 min- 1Timeout
A is the longest you'll wait for one call before giving up and freeing what you hold. Set it from measured latency: Reviews' p99 is 80 ms, so 300 ms, well below the page's own 1 s budget.
- 2Few retries, with jitter
Retry only rare, short errors (timeouts, 502/503/504, never 4xx) on safe operations. Backoff with jitter means waiting longer before each retry, by a random amount, so clients don't retry in waves. One retry, at one layer, within a 10% budget.
- 3Circuit breaker
A stops calling a service after too many failures. Closed: calls pass, failures are counted. Open (at 50% failures): no calls for 30 s, instant fallback. Half-open: a few probes; one failure reopens it.
- 4Bulkhead
A is a separate, limited pool for one dependency. Reviews may use at most 40 of the 200 workers; the 41st call gets the fallback at once. 160 workers always stay free for booking.
- 5Fallback
What guests see when the breaker is open: the hotel page without the reviews widget, or the cached average rating. A breaker without a fallback just turns a slow error into a fast one.
Watch a decision being made
4 minYou write the Reviews policy; Lina checks it
Reviews' p99 is 80 ms; hotel pages call it 150 times a second; the web service has 200 workers. A slow Reviews must never take more than hotel stars down.
- 1Set the timeout from data
300 ms: about 3–4× p99, far below the page's 1 s budget.
Why: Near the median cuts healthy calls; far above p99 lets workers pile up.
- 2Limit retries
1 retry, full jitter up to 100 ms, only on timeouts and 502/503/504, within a 10% budget.
Why: One retry hides a lost packet. More retries pile load on a struggling service.
- 3Add the breaker and the bulkhead
Open at 50% of the last 100 calls (min 20), for 30 s. At most 40 concurrent Reviews calls.
yamlreviews: timeout: 300ms retry: { maxAttempts: 2, jitter: full, budget: 10% } circuitBreaker: { window: 100, minimumCalls: 20, failureRate: 50%, openDuration: 30s } bulkhead: { maxConcurrent: 40 } fallback: hide-reviewsWhy: If every other setting is wrong, the bulkhead still keeps 160 workers for booking.
- 4Design the fallback, and alert
Show the page without reviews, or the cached rating. Alert when the breaker opens.
Why: Guests can book without stars, not from an error page. The alert stops a silent fallback hiding a broken service for days.
In real life
3 minNetflix: shields in a library
- Publicly open-sourced Hystrix, which wraps each remote call with a timeout, breaker, bulkhead and fallback
- Each team writes its own fallback
- Every service must use and update the library
Fits when fallbacks need each team's business knowledge.
Istio / Envoy: shields in a service mesh
- Publicly documented: timeouts and retries set as config on proxies next to each service
- Any language, no code change
- The proxy fails fast but can't invent a business fallback
Fits many languages, with a platform team to run it.
What goes wrong
3 minMany HTTP clients wait minutes or forever; one slow dependency holds every worker.
Explicit timeouts on every remote call, from measured p99.
Attempts multiply and land in synchronised waves.
One layer owns retries, with full jitter and a budget; others fail fast and pass the deadline down.
A 400 fails again; a retried POST /payments may charge twice.
Retry only timeouts and 502/503/504, on idempotent or keyed operations.
Trade-offs
3 min| Approach | Fits when | You pay |
|---|---|---|
| Patient: long timeouts, many retries | Rarely: batch jobs, nobody waiting | Overload becomes an outage |
| Tight timeout, 1 retry, breaker + fallback | Guest-facing requests with dependencies | Latency data to tune; a fallback per feature |
| Async: queue it, answer "pending" | Emails, invoices, partner sync | Later results; a queue and workers to run |
- Maya · Product managerHotel pages without reviews convert about 15% worse. I'd rather wait 2 seconds and show them.
- Omar · On-call SREAt 150 pages a second, 2 s is 300 workers stuck. We have 200. That's Friday again.
- Maya · Product managerThen give Reviews more workers.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Each dependency's timeout, from its p99 and the caller's budget: 300 ms for Reviews.
- Which single layer owns retries, how many (1, jittered), on which errors, within a 10% budget.
- What guests see when Reviews is off. That's a product decision, not a library default.
A slow Reviews service filled all 200 StayHub web workers and took every page down. Plan first, Reviews client only. Timeout 300 ms. One retry, full jitter up to 100 ms, only on timeouts and 502/503/504, within a 10% budget. Breaker: open at 50% failures over 100 calls (min 20), 30 s, then half-open probes. Bulkhead: 40 concurrent calls. Fallback: page without reviews or cached rating. Alert when the breaker opens. Remove gateway retries. Test with a fake Reviews that sleeps 8 s.
- Find the timeout value in the code. A default of 30 s or "none" is the Friday bug.
- List every layer that retries Reviews calls and multiply the attempts yourself. Only one should retry.
- In the 8-second test, hotel pages answer in under a second without stars, and booking is unaffected.
- It adds retries at a second layer "for resilience".
- Retries without jitter, or on 400s and unkeyed
POST /payments. - It raises the timeout or the worker pool to make a failing test pass.
Check yourself
2 minNext episode: StayHub sells a hotel room and an airport transfer as one package, the transfer partner answers "no drivers", and guests are charged for half a holiday.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You write the Reviews policy; Lina checks it
Reviews' p99 is 80 ms on a normal day. The web service has 200 workers, and hotel pages call Reviews 150 times a second. Write a policy so a slow Reviews can never take more than hotel stars down with it.
- 1Set the timeout from data
p99 is 80 ms, so 300 ms: about 3–4× p99, and far below the page's own 1 s budget.
Why: Close to the median fails healthy-but-slow calls; far above p99 lets workers pile up. The caller's budget is the ceiling.
- 2Limit retries
At most 1 retry, with full jitter (random wait up to 100 ms), only on timeouts and 502/503/504, within a 10% retry budget.
Why: One retry hides a lost packet. More retries pile load on a service that's struggling. Reviews is a read, so a repeat is harmless.
- 3Add the breaker
Open when 50% of the last 100 calls fail (minimum 20 calls). Stay open 30 s, then let a few probes through.
Why: Once Reviews is clearly broken, calling it wastes 300 ms per page and slows its recovery. Failing fast costs nothing.
- 4Build the bulkhead
Reviews calls may use at most 40 of the 200 workers; a 41st call is refused at once and gets the fallback.
Why: If every other setting is wrong, this one still guarantees 160 workers for booking, search and login.
- 5Design the fallback
Show the hotel page without the reviews widget, or with the cached average rating, and alert when the breaker opens.
Why: A breaker without a fallback just turns a slow error into a fast error. Guests can book without stars; they can't book from an error page.
// Exponential backoff with "full jitter": a random wait in [0, min(cap, base * 2^attempt)].
function backoffMs(attempt: number, base = 100, cap = 2_000): number {
return Math.random() * Math.min(cap, base * 2 ** attempt);
}
async function callWithRetry<T>(
fn: (signal: AbortSignal) => Promise<T>,
{ attempts = 2, perTryTimeoutMs = 300, deadline }: { attempts?: number; perTryTimeoutMs?: number; deadline: number },
): Promise<T> {
for (let attempt = 0; ; attempt++) {
const remaining = deadline - Date.now();
if (remaining <= 0) throw new Error("deadline exceeded");
try {
// Never wait longer than the per-try timeout OR what's left of the caller's deadline.
return await fn(AbortSignal.timeout(Math.min(perTryTimeoutMs, remaining)));
} catch (err) {
const retryable = isTimeout(err) || isStatus(err, [502, 503, 504]); // never 4xx
if (!retryable || attempt + 1 >= attempts || !retryBudget.tryAcquire()) throw err;
await sleep(backoffMs(attempt));
}
}
}| Pattern | Protects against | Typical setting | Failure mode if misused |
|---|---|---|---|
| Timeout | Waiting forever on a slow call | p99 × 2–4, below the caller's deadline | Too short: false failures; too long: workers pile up |
| Retry + backoff + jitter | Rare, short blips | 1–2 retries, full jitter, idempotent ops only | Retry storms; duplicates on unsafe calls |
| Retry budget | Retries multiplying load | Retries ≤ 10–20% of traffic | Too low: blips become errors |
| Circuit breaker | Hammering a broken dependency | Open at ~50% failures, 10–60 s | Flapping; opening on 4xx caused by clients |
| Bulkhead | One dependency starving others | Separate pool per dependency | Pools too small for normal peaks |
| Load shedding | Accepting work you can't finish | Bounded queues; 429/503 with Retry-After | Shedding important traffic first |
Interactive simulation
Visual diagram
The pricing service slows down. Follow how each protection keeps the rest of the platform working.
Analogy
- In a busy restaurantWaiter gives up after 5 minutes at the hatchIn your systemA timeout
- In a busy restaurantWaiters return at different, later timesIn your systemBackoff with jitter
- In a busy restaurantFish crossed off the menu for 15 minutesIn your systemAn open circuit breaker
- In a busy restaurantDessert station with its own cooksIn your systemA bulkhead
- In a busy restaurantHost turning guests away when fullIn your systemLoad shedding
Where the analogy breaks: a restaurant manager notices problems by eye. Software must detect them automatically, from numbers like error rates and response times.
Check your understanding
Practice ladder
Trace the breaker through an incident
Scenario Warm-upThe Reviews breaker uses: window of the last 100 calls, minimum 20 calls, open at 50% failures, open for 30 s, then half-open with probe calls (any probe failure reopens; 5 probe successes close).
Timeline: - 21:10:00 Reviews starts timing out. Of the next 40 calls, 24 fail. - 21:10:05 The failure rate in the window crosses 50%. - 21:10:35 ? - 21:10:35 The first probe times out. - 21:11:05 ? - 21:11:05–21:11:06 Five probes succeed (Reviews was rolled back at 21:10:50).
Give the breaker's state at each "?" and at the end, and what guests see on hotel pages in each phase.
Fill in the Recommendations policy
Design PracticeThe hotel page also shows "Guests also viewed", from the Recommendations service: a read, p99 150 ms, about 150 calls/s, purely optional. The page's total budget is 1 s. Lina left gaps in the policy. Fill them in and give a one-line reason for each.
recommendations:
timeout: ___
retry:
maxAttempts: ___
circuitBreaker:
failureRate: 50%
openDuration: 30s
bulkhead: { maxConcurrent: ___ }
fallback: ___A policy for checkout
Design PracticeCheckout calls three dependencies, one after another: - Availability (read, p99 60 ms), required. - Payment provider (write, p99 900 ms, supports idempotency keys), required. - Currency conversion (read, p99 50 ms) to show the total in the guest's currency; optional, rates change slowly.
The checkout request has a 5 s budget. Write a timeout, retry, breaker and fallback policy for each of the three, and say what the guest sees when each one is broken.
Which layer owns retries?
Decision StretchAfter the 40-minute outage, Karim wants one rule for retries. Two proposals:
- A: Only the app retries (it knows when the guest is waiting). The gateway and services never retry.
- B: Only the service closest to each dependency retries (for example the web service for Reviews), with jitter and a 10% budget. The app and gateway never retry automatically.
Facts: the app is released every 2 weeks and old versions stay in use for months. Services deploy several times a day. Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain how a slow pricing service can take down an entire booking platform, and which patterns you'd apply in which order to prevent it.