Skip to content
Episode 68 · The 2% that every deploy costs

Health checks and rollouts

Thursday, 11:20: booking-api deploys ten times a day, and every deploy fails about 2% of requests for 40 seconds. At 1,500 requests a second, that's over a thousand errors per deploy.

Kubernetes asks your app three questions to decide whether to wait, hide it, or restart it. Get them right and deploys become boring.

30 min advancedDesigning liveness, readiness and startup probesGraceful shutdown and connection drainingRollout pacing with maxSurge and maxUnavailablePodDisruptionBudgets and voluntary disruptionsRollbacks and revision historyHealth checks (liveness/readiness)
  1. Problem

    The booking API deploys 10 times a day across 12 pods, and nodes are drained weekly for upgrades. Guests must never see errors during any of it.

  2. Why the simple system fails

    Naive probes restart healthy pods during a database blip, pods killed mid-request drop bookings, an aggressive rollout removes half the capacity at once, and a node drain evicts every replica together.

  3. Engineering concept

    Separate the questions (am I alive? can I serve? have I finished booting?), drain before dying, bound how much capacity a rollout or drain may remove, and keep the previous version one command away.

  4. Technology

    livenessProbe, readinessProbe, startupProbe, preStop + SIGTERM handling, terminationGracePeriodSeconds, maxSurge/maxUnavailable, minReadySeconds, progressDeadlineSeconds, PodDisruptionBudget, kubectl rollout undo.

  5. Trade-offs

    Safer rollouts are slower and need spare capacity; sensitive probes detect failure fast but cause flapping and restarts; strict PDBs protect availability but can block node maintenance.

01

The problem

2 min
Thursday, 11:20. The third deploy of booking-api today shows the same spike.
  1. Omar · On-call SRE
    Every deploy: ~2% errors for ~40 s. booking-api runs 12 pods at about 1,500 requests a second.
  2. Maya · Product manager
    So each deploy costs more than a thousand failed requests. Some are checkouts.
  3. Lina · Senior engineer, your mentor
    You lead this one. Guess first.
02

The big idea

4 min
Three questions Kubernetes asks your app
The big idea

Ready before traffic, drained before exit. Each probe answers one question, and triggers one action.

Deploying less often isn't the fix: ten small deploys a day is a feature. Fix the arrival and the departure instead, and deploys become boring.

03

How it works

5 min
A pod's life during a rollout, done right
  1. 1
    Boot

    A asks "have you finished booting?" It gives booking-api up to 150 s; nothing else is checked until it passes.

  2. 2
    Join

    A asks "can you take traffic now?" If not, the pod is hidden from the Service, not restarted. /readyz says yes only once the cache is warm (~40 s).

  3. 3
    Serve

    A asks "are you stuck for good?" If not answering, the container is restarted. /livez checks only that the event loop is alive.

  4. 4
    Leave

    On SIGTERM, a 5 s preStop pause. Endpoint removal runs in parallel with SIGTERM, so traffic still arrives for a few seconds.

  5. 5
    Drain and exit

    Graceful shutdown: stop accepting, finish in-flight requests, close database connections, exit well within the 45 s grace period.

  6. 6
    Pace the rollout

    maxSurge: 25% adds up to 3 extra pods (15 total); maxUnavailable: 0 keeps at least 12 ready. Capacity never dips.

04

Watch a decision being made

4 min
Worked example

You make booking-api deploys invisible

Every deploy fails ~2% of requests for ~40 s. Make deploys error-free without slowing them more than necessary. Lina pushes back in the "why" lines.

  1. 1Measure the warm-up

    A fresh pod: process up at 3 s, cache warm at about 40 s. Worst case seen: 90 s on a busy node.

    Why: Probe timings are maths on these numbers. Guessing them is how crash loops start.

  2. 2Write the endpoints and probes

    /livez: 200 if the event loop answers. /readyz: 200 only if the cache is warm and the pod isn't shutting down or overloaded. Startup probe every 5 s, up to 30 failures (150 s).

    ts
    if (req.url === "/readyz") {
  const ok = !shuttingDown && cacheWarm() && inFlight < 200;
  res.statusCode = ok ? 200 : 503;
  return void res.end();
}

    Why: "Readiness on local signals only. If it checked the shared database, a blip would empty the Service of every pod at once."

  3. 3Leave politely, and pace the rollout

    SIGTERM → readiness fails, server.close(), close the pool, exit. preStop: sleep 5, grace 45 s. maxSurge: 25%, maxUnavailable: 0, minReadySeconds: 10.

    Why: The pause covers the parallel endpoint removal; a pod crashing 5 s after ready doesn't count as done. Needs room for 3 extra pods.

  4. 4Prove it under load

    Deploy during a 1,500 requests/s load test, then kill the database in staging and check that no pod restarts.

    Why: Each test targets one failure you designed for. Result: about 5 failed requests per deploy, down from about 1,200.

The whole thing in one line: Three separate questions (booted? ready? stuck?), a polite exit and a paced rollout turned each deploy into a non-event, at the cost of ~5 minutes per deploy and room for 3 extra pods.
05

In real life

3 min
Probes stop pods that can't start. What stops a bad release?

Netflix: automated canary analysis

  • Has publicly described Spinnaker and Kayenta: a new version gets a small slice of traffic
  • Its metrics are compared with the old version automatically
  • Needs traffic splitting and solid metrics

Worth it for huge traffic where a bad minute is very expensive.

Etsy: deploy often, roll back fast

  • Has publicly described deploying many times a day with a simple one-button tool
  • Small changes, heavy monitoring, quick rollback
  • A bad change reaches everyone for a few minutes

Worth it when changes are small and monitoring catches problems fast.

06

What goes wrong

3 min
Liveness checks a shared dependency

One blip fails every pod at once, and they all restart cold.

Instead

Liveness checks only the process. Ask: would a restart fix this?

Exiting immediately on SIGTERM

Traffic still arrives for a few seconds; in-flight requests are cut.

Instead

preStop pause, then stop accepting, finish in-flight work, exit before the grace period.

Treating `rollout undo` as the plan for data changes

Rollback restores old code, not the old schema or data the new version wrote.

Instead

Backward-compatible migrations (expand, then contract).

07

Trade-offs

3 min
Settings (12 replicas)Pods during rolloutEffect
Surge 25%, unavailable 25%Up to 15; at least 9 readyFast; loses a quarter of capacity
Surge 25%, unavailable 0Up to 15; at least 12 readyNo capacity loss; room for 3 extra pods
Surge 50%, unavailable 0Up to 18; at least 12 readyFaster, still safe; room for 6 extra pods
Friday, 10:00. Each deploy now takes about 5 minutes.
  1. Karim · CTO
    Deploys went from 90 seconds to 5 minutes. Ten a day. Use 50% surge and 25% unavailable.
  2. Omar · On-call SRE
    25% unavailable is 3 of 12 pods gone at peak. The other 9 run hot, fail readiness, and we're back to errors.
  3. Karim · CTO
    Then deploy outside peak hours.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • What each probe may check: liveness = "is this process stuck?" only; readiness = this pod's own ability to serve; never the shared database.
  • Rollout pacing for 12 replicas at ~1,500 req/s: maxSurge: 25%, maxUnavailable: 0, so capacity never drops below 12.
  • The trade you accept: ~5-minute deploys instead of ~1.5, for ~0 errors per deploy.
Tell Claude Code
Fix booking-api's health checks: every deploy fails ~2% of requests for 40 s. I want a startup probe allowing up to 150 s; /livez checking only the process, no database; /readyz on local signals like cache warmed; preStop sleep 5 s; SIGTERM handling that finishes in-flight requests within the 45 s grace period; maxSurge 25%, maxUnavailable 0, minReadySeconds 10. Plan first. Then in staging, deploy under load and run a database failover, and show me the error rate during each.
Check before you accept
  • Read every probe handler: none touches PostgreSQL, Redis or another service.
  • The deploy-under-load result: errors about zero, not 2% for 40 s.
  • The failover drill: pods keep running (maybe returning errors), not all restarting at once.
Red flags
  • It adds the database to the liveness probe "to be thorough".
  • It "fixes" crash loops by turning liveness off, or a huge initialDelaySeconds instead of a startup probe.
  • It claims zero-downtime deploys without having deployed under load.
09

Check yourself

2 min
Q1/3Reasoning
A Deployment has 10 replicas with maxSurge: 25% and maxUnavailable: 25%. During the rollout, what is the minimum number of available pods Kubernetes guarantees?
pods
How confident are you?

Next episode: a three-person team wants its own Kubernetes cluster for a brand-new product. You lead the decision.

Finished?

Mark it complete to update your skill map.

Next: When not to use Kubernetes
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Pacing a rollout with 12 pods

You lead the fix. Lina's comments are in the "why" lines where she pushed back.

Worked example

You make booking-api deploys invisible

Every booking-api deploy fails ~2% of requests for ~40 s. Make deploys and weekly node upgrades error-free without slowing them down more than necessary.

  1. 1Measure the warm-up

    Time a fresh pod: process up at 3 s, pricing rules loaded and cache warm at about 40 s. Worst case seen: 90 s on a busy node.

    Why: Probe timings are maths on these numbers. Guessing them is how crash loops start.

  2. 2Write three endpoints

    /livez: returns 200 if the event loop answers. /readyz: 200 only if rules are loaded, the cache is warm, and the pod isn't shutting down or overloaded.

    ts
    if (req.url === "/livez") return void res.end("ok");
if (req.url === "/readyz") {
  const ok = !shuttingDown && cacheWarm() && inFlight < 200;
  res.statusCode = ok ? 200 : 503;
  return void res.end();
}

    Why: Lina: "readiness on local signals only. If it checked the shared database, a database blip would empty the Service of every pod at once."

  3. 3Set the probes

    Startup probe on /livez: every 5 s, up to 30 failures (150 s). Liveness on /livez: every 10 s, 3 failures. Readiness on /readyz: every 5 s, 2 failures.

    Why: The startup budget (150 s) covers the 90 s worst case with room. Detection time is period × threshold: about 30 s to restart a truly stuck pod.

  4. 4Leave politely

    On SIGTERM: mark shutting down (readiness fails), server.close() to finish in-flight requests, close the pool, exit. Add preStop: sleep 5 and terminationGracePeriodSeconds: 45.

    Why: Endpoint removal runs in parallel with SIGTERM. The 5 s pause covers it; the drain must end before the 45 s grace period or the pod is killed anyway.

  5. 5Pace the rollout

    maxSurge: 25%, maxUnavailable: 0, minReadySeconds: 10, progressDeadlineSeconds: 300. Add a PodDisruptionBudget with maxUnavailable: 1 for node upgrades.

    Why: Capacity never dips below 12 ready pods; a pod that crashes 5 s after becoming ready doesn't count; a stuck rollout is marked failed instead of hanging. Needs room for 3 extra pods.

  6. 6Prove it

    Deploy during a load test at 1,500 requests/s and watch the error rate. Then drain a node. Then kill the database in staging and check that no pod restarts.

    Why: Each test targets one failure mode you just designed for. Result: about 5 failed requests per deploy, down from about 1,200.

The whole thing in one line: You separated the three questions (booted? ready? stuck?), made pods drain before exiting, and paced the rollout so capacity never dips, turning a deploy from ~1,200 failed requests into a non-event at the cost of ~5 minutes per deploy and room for 3 extra pods.
SettingRollout with 12 replicasEffect
maxSurge 25%, maxUnavailable 25%Up to 15 pods; at least 9 availableFast; temporarily loses 25% of capacity
maxSurge 25%, maxUnavailable 0Up to 15 pods; at least 12 availableNo capacity loss; needs room for 3 extra pods
maxSurge 0, maxUnavailable 1At most 12 pods; at least 11 availableNo extra nodes needed; slow, one pod at a time
maxSurge 100%, maxUnavailable 0Up to 24 podsNear blue-green: whole new set first; doubles capacity briefly
Percentages round: maxSurge rounds up, maxUnavailable rounds down. Both can't be 0.
11

Interactive simulation

12

Visual diagram

Click a component

One rolling-update step with maxSurge 1 and maxUnavailable 0, including graceful shutdown of the old pod. A small 4-pod example; booking-api runs 12 in summer.

13

Analogy

  • A hotel front deskNew receptionist still logging inKubernetesStartup probe: don't judge yet
  • A hotel front deskFlipping the desk sign to "closed"KubernetesReadiness: no traffic for now
  • A hotel front deskReplacing someone who collapsedKubernetesLiveness: restart the container
  • A hotel front deskFinish the guest in front, then leaveKubernetesGraceful shutdown
  • A hotel front deskNever fewer than 10 desks openKubernetesA PodDisruptionBudget

Where the analogy breaks: you'd never fire a receptionist because the booking system is down. A badly written liveness check does exactly that, to every pod.

14

Check your understanding

Q1/3Debugging
A new version takes ~60 s to boot. Pods now crash-loop. Given this config, why?
livenessProbe:
  httpGet: { path: /livez, port: 3000 }
  periodSeconds: 10
  failureThreshold: 3
# no startupProbe, no initialDelaySeconds
How confident are you?
15

Practice ladder

Step 1 · Trace

Trace one rollout cycle

Scenario Warm-up

search-api has 8 replicas, maxSurge: 25%, maxUnavailable: 0, minReadySeconds: 10. New pods take 20 s to become ready. A pod of the new version crashes 5 s after becoming ready (a bug that only shows under traffic).

Trace the rollout: how many pods are created first, when old pods are removed, and what happens when the new pods crash.

Step 2 · Complete

Put the graceful shutdown in order

Scenario Practice

During a rollout, Kubernetes terminates an old booking-api pod. Put these events in order and fill in the missing step X:

  • A. The process exits with code 0
  • B. Kubernetes sends SIGTERM to the container (after preStop)
  • C. The pod is marked Terminating; endpoint removal starts and the preStop hook (sleep 5) runs
  • D. The app sets shuttingDown = true and /readyz returns 503
  • E. The app closes its database pool
  • X. ___

Also: what happens if the app hasn't exited 45 s after termination started?

Step 3 · Solve

The new version crash-loops

Debugging Practice

pricing-api 4.0 loads a 2 GB pricing model at start: about 60 s on a quiet node, up to 100 s on a busy one. Its probes:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5

/livez only answers once the model is loaded. Pods are now in CrashLoopBackOff. Explain why and fix it.

Step 4 · Decide

Should readiness check the database?

Decision Stretch

The hotel-page service (8 pods) shows hotel details. Hotel details are cached in each pod for 5 minutes; about 90% of requests hit the cache. A teammate proposes adding a database check to /readyz "so we don't serve when the database is down".

Facts: the database fails over about once a month for ~40 s. Without the check, during a failover, cached pages still load and the 10% of uncached pages fail with a friendly error.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Walk through what happens, step by step, when you deploy a new version of a 4-pod service with maxSurge 1 and maxUnavailable 0, and how you'd make sure no guest request fails.

0 words