Skip to content
Episode 59 · Nine minutes on a Friday

Deployment strategies: recreate, rolling, blue-green, canary, shadow

Friday, 18:00: a deploy replaces all twelve booking servers at once, and checkout is down until 18:09, the busiest hour of the week. The code change itself was correct.

Some releases will be bad. Here are five ways to swap old code for new so a bad one stays small and short.

30 min intermediateRecreate, rolling, blue-green, canary, shadowBlast radius and cohort comparisonRollback vs roll-forwardExpand/contract database migrationsRolling updatesHealth checks (liveness/readiness)
  1. Problem

    The hotel booking API ships 10 times a week. Every deploy is a moment when old code stops and new code starts, and roughly one release in twenty has a bug the tests missed.

  2. Why the simple system fails

    Stopping everything and starting the new version causes downtime; replacing everything at once means a bad release hits 100% of guests before anyone notices; and a schema change can make rolling back impossible.

  3. Engineering concept

    Control exposure: decide how much traffic the new version gets, when, how you detect a problem, and how fast you can undo it. Keep the database compatible with both versions.

  4. Technology

    Load balancers and health checks, rolling updates, blue-green environments, canary routing with automated analysis, traffic mirroring, expand/contract migrations.

  5. Trade-offs

    Smaller blast radius and faster rollback cost extra capacity, slower rollouts, routing infrastructure, and the discipline of running two versions side by side.

01

The problem

2 min
Friday, 18:09. The busiest hour of the week.
  1. Omar · On-call SRE
    The 18:00 deploy stopped all 12 booking instances at once. Ready again at 18:09: about 165 bookings lost.
  2. Omar · On-call SRE
    A startup migration renamed guest_name to guest_full_name on a 40-million-row table and locked it.
  3. You · Engineer at StayHub
    Why not replace them a few at a time?
02

The big idea

4 min
Why everything had to stop
The big idea

Change the database in steps both versions understand; then any rollout is possible.

"Never deploy at 18:00 on a Friday" only moves the outage to a quieter hour. Every deploy answers one question: how does traffic move from old code to new?

03

How it works

5 min

Each strategy trades downtime, rollback speed, cost and : how many guests a bad release reaches.

Ways to swap running code
  1. 1
    Recreate

    Stop all, start new. Downtime every time; a bad release hits 100%. Friday's choice.

  2. 2
    Rolling update

    A replaces a few servers at a time; old and new run together. No downtime, but no comparison: a quiet bug rolls to 100%.

  3. 3
    Blue-green

    runs a full second copy, then flips all traffic. Undo in seconds, but 100% of guests get the new version at once.

  4. 4
    Canary

    A sends a small slice (5%) of real traffic to the new version, compares it with stable, then 25%, then 100%. A bad release reaches few guests.

  5. 5
    Expand / contract

    changes the database in releases: add the new column, write both and backfill, read the new one, drop the old one later. Every step works for old and new code.

04

Watch a decision being made

4 min
Worked example

You plan the rename properly; Lina checks

Rename the column on a 40-million-row table with zero downtime, and roll out so a bad release reaches at most 5% of guests.

  1. 1Split the schema change

    Release 1: add the column (instant). 2: write both, backfill in batches of 10,000. 3: read the new one. 4: drop the old.

    sql
    -- Release 1 (EXPAND)
ALTER TABLE bookings ADD COLUMN guest_full_name text;
-- Release 2: backfill in small batches
UPDATE bookings SET guest_full_name = guest_name
WHERE id BETWEEN $1 AND $2 AND guest_full_name IS NULL;
-- Release 4 (CONTRACT), only when no old code remains
ALTER TABLE bookings DROP COLUMN guest_name;

    Why: Running code always finds its columns; rollback never meets a missing one.

  2. 2Run migrations outside app startup

    A separate pipeline step with SET lock_timeout = '3s'.

    Why: Friday's rewrite at boot froze every instance; a time-limited step fails fast instead.

  3. 3Choose canary steps and define "worse"

    5% for 10 min, 25% for 15 min, 100%. Fail if 5xx > 2× stable, p95 > 1.5× stable, or bookings/min per instance < 90% of stable. At least 500 canary requests per step.

    Why: At ~1,000 req/min, 5% gives 500 requests in 10 minutes. A business metric catches quiet bugs.

  4. 4Rehearse the rollback

    On a quiet morning, deploy release 2, roll back to 1, confirm nothing is lost.

    Why: A rollback never run is a hope.

The whole thing in one line: Four backward-compatible releases, migrations outside startup, a canary judged on what guests feel, and a rehearsed rollback.
05

In real life

3 min
Instant flip versus small slice

AWS Elastic Beanstalk: blue/green

  • Publicly documents deploying to a second environment, then swapping URLs
  • Rolling back is swapping back: seconds
  • At the swap, every user moves at once

Fits when instant, complete rollback matters most.

Netflix: automated canary analysis

  • Has publicly described Kayenta (open-sourced with Google) scoring canaries in Spinnaker
  • New versions are compared with a baseline on many metrics
  • Needs good metrics and routing

Fits frequent releases where quiet bugs are the main risk.

06

What goes wrong

3 min
Relying on health checks to catch bad releases

Costly bugs return 200 OK.

Instead

Compare with stable on errors, latency and a business metric (average tax per booking).

Renaming a column in the same release as the code

Old code breaks at once; rollback becomes impossible.

Instead

Expand/contract over several releases.

Judging a canary on a handful of requests

One timeout looks like 10% errors; a real bug can show none.

Instead

A minimum sample per step: wait for traffic, not just time.

07

Trade-offs

3 min
StrategyBlast radiusRollbackExtra capacity
Recreate100%, with downtimeRedeploy (minutes)None
RollingGrows to 100%Another rollout+1 to +25%
Blue-green100% at the flipFlip back (seconds)+100% during switch
Canary1–25% until promotedWeight to 0 (seconds)+5 to +25%
Wednesday, 14:00. Choosing the booking rollout.
  1. Karim · CTO
    Blue-green. Flip, and if anything's wrong, flip back in five seconds.
  2. Omar · On-call SRE
    Five seconds after someone notices. The tax bug went unnoticed for three hours, on 100% of guests.
  3. Karim · CTO
    A canary needs per-version metrics, a router, rules. That's weeks.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • Which strategy each service uses: canary for Booking, not the old recreate script.
  • What the canary is judged on: 5xx, p95 and a business signal. At least 500 requests per step.
  • How schema changes ship: expand/contract, never a rename in the same release as its code.
Tell Claude Code
We need to replace bookings.guest_name with guest_full_name in the Booking service. Last time a one-step rename took checkout down for 9 minutes. Plan expand/contract across separate releases: add the column, dual-write, backfill the 40-million-row table in small batches without long locks, switch reads, stop writing the old column, then drop it. Each release must work with the one before it. Give me a numbered list of releases with the migration, code change and rollback for each. No code yet. Deploys go through the canary: 5%, 25%, 100%.
Check before you accept
  • Claude says, for each release, that it works with the previous version.
  • No migration drops or renames a column alongside the code that depends on it.
  • The backfill says its batch size, duration and locks.
Red flags
  • A destructive migration at app startup.
  • "We can always roll back" while dropping or renaming a column.
  • Judging a release on readiness checks and status codes only.
09

Check yourself

2 min
Q1/3Scenario
A release miscalculates the tourist tax on 10% of bookings. Responses are 200 OK and every instance passes its readiness probe. Which setup would most likely limit the damage?
How confident are you?

Next episode: the new cancellation flow has been merged and deployed for a month, and Maya still can't switch it on for just one hotel.

Finished?

Mark it complete to update your skill map.

Next: Feature flags: separate deploying code from releasing features
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

A canary that knows what "worse" means
Worked example

You plan the rename properly; Lina checks

Rename bookings.guest_name to guest_full_name on a 40-million-row table, with zero downtime, and roll the booking service out so a bad release reaches at most 5% of guests.

  1. 1Split the schema change

    Release 1: ADD COLUMN guest_full_name text (nullable, instant). Release 2: write both columns; backfill in batches of 10,000 rows. Release 3: read the new column. Release 4: drop the old one.

    Why: At every moment, the running code (old or new) finds the columns it needs. Rollback never meets a missing column.

  2. 2Run migrations outside app startup

    Migrations run as their own pipeline step before the deploy, with a lock timeout (SET lock_timeout = '3s'), never inside each instance's boot.

    Why: Friday's 40M-row rewrite at boot held a lock and delayed every instance. A separate, time-limited step fails fast instead of freezing checkout.

  3. 3Choose canary steps

    5% for 10 minutes, then 25% for 15 minutes, then 100%, each step gated by analysis against the stable version.

    Why: Small first step, enough time and traffic to judge. At ~1,000 req/min, 5% gives 500 requests in 10 minutes.

  4. 4Define "worse"

    Fail if 5xx ratio > 2× stable, p95 > 1.5× stable, or bookings per minute per instance < 90% of stable; require ≥ 500 canary requests per step.

    Why: Health checks catch crashes, not wrong answers. A business metric catches the quiet bugs; a minimum sample avoids judging on noise.

  5. 5Rehearse the rollback

    On a quiet morning, deploy release 2, roll back to release 1, and confirm the app works and no data is lost.

    Why: A rollback that's never been run is a hope. Lina's check: also confirm the backfill job stops cleanly when rolled back.

The whole thing in one line: You split the rename into four backward-compatible releases, moved the migration out of startup with a lock timeout, rolled the service out as a canary judged on errors, latency and bookings per minute, and rehearsed the rollback.
sql
-- Release 1 (EXPAND): instant, nullable, no table rewrite
ALTER TABLE bookings ADD COLUMN guest_full_name text;
-- app v1 writes BOTH columns, still reads guest_name

-- Release 2 (MIGRATE): backfill in small batches, short locks
UPDATE bookings SET guest_full_name = guest_name
WHERE id BETWEEN $1 AND $2 AND guest_full_name IS NULL;
-- app v2 reads guest_full_name, still writes both

-- Release 3: app v3 stops writing guest_name (wait until no v2 instance remains)

-- Release 4 (CONTRACT): only now remove the old shape
ALTER TABLE bookings DROP COLUMN guest_name;
Expand/contract across four releases. Each one can be rolled back without touching the schema.
11

Interactive simulation

12

Visual diagram

Click a component

A canary release of booking-api v42 with automated analysis. Click components or start the walkthrough.

13

Analogy

  • Changing a restaurant's menuClose for an hour, reopen with the new menuDeploying new codeRecreate: brief downtime
  • Changing a restaurant's menuSwap menus table by tableDeploying new codeRolling: two versions run together
  • Changing a restaurant's menuOpen a second dining room, move everyoneDeploying new codeBlue-green: flip all traffic
  • Changing a restaurant's menuGive the new menu to one table in twentyDeploying new codeCanary: a small slice of users
  • Changing a restaurant's menuCook every order twice, chef tastes the copyDeploying new codeShadow: copied traffic, hidden answers
  • Changing a restaurant's menuReorganise the pantry so both menus workDeploying new codeDatabase changes both versions can use

Where the analogy breaks: a guest can tell you the dish is bad. Software users mostly just leave. That's why you need numbers, not complaints, to judge a release.

14

Check your understanding

Q1/3Reasoning
Order the steps to safely replace bookings.price (integer euros) with bookings.price_cents while deploying with rolling updates:
  1. 1.Drop the price column
  2. 2.Deploy code that reads price_cents (still writes both)
  3. 3.Add nullable price_cents column
  4. 4.Deploy code that stops writing price
  5. 5.Deploy code that writes both columns, still reads price
  6. 6.Backfill price_cents for old rows in batches
How confident are you?
15

Practice ladder

Step 1 · Trace

Trace a canary at a quiet hour

Estimation Warm-up

The booking service gets 1,000 requests per minute at peak and 200 per minute at 07:00. The canary's first step is 5%, and the analysis needs at least 500 canary requests.

  1. How long must the first step last at peak, and at 07:00?
  2. A release with a bug that fails 4% of requests goes out at 07:00 with a fixed 10-minute first step. How many canary requests and failures does the analysis see? Can it judge?
  3. What rule would you change?
Step 2 · Complete

Order the price_cents migration

Design Practice

StayHub wants to replace bookings.price (whole euros) with bookings.price_cents, deployed with canary releases. Lina's plan has gaps. Fill in releases 2 and 4.

  • Release 1: ADD COLUMN price_cents integer (nullable). App writes price and price_cents; reads price.
  • Release 2: ___
  • Release 3: App stops writing price. Wait until no release-2 instance is running.
  • Release 4: ___

For each, say why it's safe if the release is rolled back.

Step 3 · Solve

Roll out the new cancellation fees

Design Practice

A release changes how cancellation fees are computed:

  • Code changes in the booking service (12 instances, ~1,000 requests/min at peak).
  • A new column cancellation_policy_version on bookings.
  • A wrong fee doesn't error: the guest just pays too much or too little.
  • About 300 cancellations happen per hour.

Design the rollout: migration steps, strategy and steps, what the analysis compares, and how you'd undo it.

Step 4 · Decide

Ban Friday deploys?

Decision Stretch

After the 18:00 outage, Karim proposes: no deploys after 15:00 on Fridays, and none on weekends.

Facts: ~10 releases a week; Friday afternoon and Sunday evening are the two busiest booking periods; the booking service now has expand/contract migrations and canary releases with automatic rollback; on-call is staffed all weekend.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain to a teammate why your team uses canary releases and expand/contract migrations instead of just running a rolling update after the migration script.

0 words