Skip to content
Episode 158 · Friday, 17:30

Deploy and operate what the agent built

Friday, 17:30. Maya's room-upgrade offers deploy to every guest. 17:34: checkout errors jump from 0.2% to 9%. 17:50: someone rolls back, and errors climb to 30%. Nobody planned how to undo it.

Shipping isn't the finish line. Know how to switch it on slowly, how to undo it, and which numbers to watch.

30 min intermediateDeploy readiness (rollback plan before deploy)Post-deploy verification (watch the numbers)Cost guardrails for what you runDev, staging & production environmentsRollback vs roll-forwardExpand/contract database migrations
  1. Problem

    An agent-built feature deploys on a Friday evening. Four minutes later, 9% of checkouts fail, and nobody knows how to undo it.

  2. Why the simple system fails

    The agent changed code, configuration and the database schema together; production was missing a setting staging had; the rollback brought back code that no longer matched the database. Nobody was watching the right number.

  3. Engineering concept

    A deploy is ready when you know what it needs in each environment, how to undo it (including data), how it's switched on gradually, and which numbers prove it works. Operating means watching those numbers, keeping restorable backups and capping costs.

  4. Technology

    Environments and environment variables, CI and staging smoke tests, feature flags, backward-compatible (expand/contract) migrations, rollback plans, post-deploy checks, symptom alerts, backups with restore drills, budget and spend limits.

  5. Trade-offs

    Readiness checks and gradual rollouts slow each release; skipping them is fast until a bad deploy reaches every guest. The safest time to deploy is when someone can watch and undo, not a day of the week.

01

The problem

2 min
Friday, 17:30. Room-upgrade offers deploy to every guest. StayHub handles ~2,200 checkouts an hour on Friday evenings.
  1. Omar · On-call SRE
    17:34: checkout errors from 0.2% to 9%. I saw it at 17:41 because support pinged me.
  2. Maya · Product manager
    I rolled back at 17:50. It got worse: 30%.
  3. Omar · On-call SRE
    Fixed forward at 18:20. About 380 guests couldn't pay.
02

The big idea

4 min
The deploy nobody could undo
The big idea

Plan every deploy so a mistake reaches few guests, is noticed by a number, and can be undone in seconds.

The code was fine. What failed was everything around it: a missing setting, a database rename, no switch, and alerts on CPU instead of checkout. Four questions catch all of that.

03

How it works

5 min

A chef doesn't put a new dish on every table on the busiest night, and doesn't change the plates the old dishes use the same evening. A few tables first, watch the plates come back, keep the old menu ready.

Questions before any deploy
  1. 1
    What does it need?

    An is a setting given by where the code runs; each environment has its own. Diff the names: staging vs production. Plus secrets and migrations.

  2. 2
    How do we undo it?

    A says how, how long, and what happens to data already written. Rollback rewinds code, not data, so migrations follow : add now, remove in a later release.

  3. 3
    How is it switched on?

    A turns a feature on for some guests without a deploy. Deploy dark, then 5% → 25% → 100%, 30 minutes each.

  4. 4
    What proves it works?

    The number guests feel, with a baseline and a stop rule decided in advance: checkout success 99.8%, p95 1.9 s; flag off below 99.5% or above 2.5 s for 5 minutes.

  5. 5
    Who's watching?

    Someone who can read those numbers and flip the flag, for the first hour.

  6. 6
    What can it spend?

    Anything calling a paid API gets retry limits, a budget alert, a spend cap and its own key.

04

Watch a decision being made

4 min
Worked example

You write the redeploy plan; Lina only reads it

Ship upgrade offers again so a mistake reaches few guests, is undone in seconds and is noticed by a number.

  1. 1List what it needs

    "List every variable, secret and migration this feature needs." Then diff names between environments: UPGRADE_PRICING_URL exists only in staging.

    bash
    diff <(platform env list --env staging | cut -d= -f1 | sort) \
     <(platform env list --env production | cut -d= -f1 | sort)

    Why: That one line was Friday's 9%. It goes into production first.

  2. 2Additive migration, behind a flag

    "Only add price_cents, keep price, rename nothing. Wrap the offer in upgrade_offers, default off, with a test that checkout is unchanged when off."

    Why: The previous version still runs on the new database, and undo becomes a flag flip.

  3. 3Numbers and stop rule in advance

    Baselines: 99.8% success, p95 1.9 s. Flag off if flagged guests drop below 99.5% or exceed 2.5 s for 5 minutes. Alert on checkout success, not only CPU.

    Why: Thresholds decided in advance stop arguments with the graph at 18:00.

  4. 4Rehearse, ramp, leave a runbook

    Rehearse flag-off and rollback on staging. Tuesday 10:00: 5%, 25%, 100%, 30 minutes each; you watch until 11:30. Claude drafts a one-page runbook.

    Why: A rollback never run is a hope; the runbook lets on-call act at 03:00.

The whole thing in one line: Missing setting found first, data change compatible with old code, deploy separated from release, numbers decided, undo rehearsed: the feature can fail small and anyone can stop it.
05

In real life

3 min
Prevent the spread, or undo fast

Netflix: automated canary analysis

  • New version gets a slice of traffic
  • Metrics compared with the current version
  • Promoted or rolled back automatically

Stop a bad version reaching everyone.

Vercel / Heroku-style instant rollback

  • Each deploy kept as an immutable version
  • Rollback = point traffic back
  • Seconds, not minutes

Make undo cheap.

06

What goes wrong

3 min
Assuming production has what staging has

Agents add settings where they test.

Instead

Diff variable names before each deploy; list new settings in the plan.

Alerts on CPU only

Checkout can fail on healthy machines.

Instead

Alert on what guests feel: checkout success, latency.

Paid API calls without limits

A loop runs all weekend and can starve shared quotas.

Instead

Retry limits with backoff, budget alerts, spend caps, a separate key per job.

07

Trade-offs

3 min
Release styleFits whenYou pay
Flag + gradual rolloutCheckout, pricing, bookings, guest emailsFlag code to remove; slower to 100%
Deploy + instant rollbackInternal tools, low-risk changesEveryone exposed until noticed
Big-bang releasePrototypes with no real usersAll guests at once; undo may be impossible
Monday, 11:00. Retro on Friday.
  1. Karim · CTO
    Simple rule: no deploys on Fridays. Ever.
  2. Omar · On-call SRE
    Then Thursday becomes the new Friday. Nobody could watch, there was no flag, the rollback didn't work.
  3. Karim · CTO
    Fridays are when people leave early. That's real.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • Who watches and the stop rules: below 99.5% success or above 2.5 s p95 for 5 minutes means flag off.
  • Which migrations a release may contain: additive only; drops and renames in a later release.
  • What a feature may spend: budget alerts, spend caps, its own key or quota.
Tell Claude Code
Prepare the redeploy of room-upgrade offers; don't deploy. Write docs/deploys/upgrade-offers.md listing every variable, secret and migration needed. Rewrite the migration to only add columns. Put the offer behind upgrade_offers, default off, with a test that checkout is unchanged when off. Include rollback steps and timing, metrics with baselines (99.8%, p95 1.9 s), stop rules, and variables in staging but not production.
Check before you accept
  • Nothing exists only in staging; diff the variable names yourself.
  • The migration only adds; no rename, no drop, no NOT NULL without a default.
  • Flag-off and rollback rehearsed on staging, with timings noted.
Red flags
  • A rename or drop in the same release as the code change.
  • The plan turns the feature on for everyone at once, or has no rollback section.
  • The agent asks for production deploy credentials.
09

Check yourself

2 min
Q1/3Debugging
A feature works on staging and crashes in production with "UPGRADE_PRICING_URL is not defined". What's the most likely cause?
How confident are you?

Next episode: Claude Code's pull request to auto-refund cancelled bookings is 40 lines and all green. It touches the payment service. Karim asks: "Who reviewed this?"

Finished?

Mark it complete to update your skill map.

Next: When to ask a human: knowing your limits
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

A plan you can check
Worked example

Your redeploy plan for upgrade offers (Lina only reads it)

Ship room-upgrade offers again, this time so that a mistake reaches few guests, can be undone in seconds, and is noticed by a number, not by support.

  1. 1List what it needs

    Ask Claude Code: "List every environment variable, secret, service and migration this feature needs." Then diff variable names between staging and production.

    bash
    diff <(platform env list --env staging | cut -d= -f1 | sort) \
     <(platform env list --env production | cut -d= -f1 | sort)

    Why: UPGRADE_PRICING_URL shows up only in staging. That one line was Friday's 9%. It's added to production before anything else.

  2. 2Make the migration additive

    "Rewrite the migration so it only adds price_cents, keeps price in sync, and doesn't rename anything. Dropping price will be a separate release next month."

    Why: With only additions, the previous version still runs against the new database, so redeploying it is a real undo.

  3. 3Put it behind a flag

    "Wrap the upgrade offer in the upgrade_offers flag, default off. With the flag off, checkout must behave exactly as before; add a test for that."

    Why: Deploying and releasing become two separate moments. Undo becomes a flag flip, faster than any redeploy.

  4. 4Name the numbers and the stop rule

    Baselines from last Tuesday: checkout success 99.8%, p95 1.9 s. Stop rule: flag off if success for flagged guests drops below 99.5% or p95 exceeds 2.5 s for 5 minutes. Add an alert on checkout success, not only CPU.

    Why: Deciding the thresholds in advance stops anyone from arguing with the graph at 18:00.

  5. 5Rehearse, then ramp with someone watching

    Rehearse flag-off and rollback on staging. Tuesday 10:00: deploy dark, 5% for 30 min, 25% for 30 min, 100%. You watch until 11:30.

    Why: A rollback you've never run is a hope. A weekday morning with a person watching is what made it safe, not the day itself.

  6. 6Leave a runbook

    Ask Claude Code for a one-page runbook: what the feature does, its flag, its dashboard, the stop rule, and who owns it.

    Why: Whoever is on call on Saturday at 03:00 can act without having been in this meeting.

The whole thing in one line: You found the missing setting before deploying, made the data change compatible with the old code, separated deploy from release with a flag, decided the numbers and stop rule in advance, rehearsed the undo, and left a runbook, so the feature can fail small and be stopped by anyone.
SignalBaselineStop rule (5 min)
Checkout success, flagged guests99.8%< 99.5%
Checkout p95 latency1.9 s> 2.5 s
Errors tagged "upgrade"0> 10 per minute
Upgrades sold per hour (at 25%)unknown≈ 0 after 1 hour: investigate, don't ramp
11

Visual diagram

Click a component

From the agent's branch to a feature you operate. Click a component, or follow the walkthrough.

12

Analogy

  • A restaurant's new dishEvery ingredient in the kitchenYour releaseEvery setting in production
  • A restaurant's new dishOffer it to a few tables firstYour releaseFeature flag at 5%
  • A restaurant's new dishWatch the plates that come backYour releaseWatch checkout success and errors
  • A restaurant's new dishOld menu ready to reprintYour releaseA tested undo plan
  • A restaurant's new dishSame plates for old dishesYour releaseDatabase changes old code still fits

Where it breaks: a dish served to a few tables is noticed quickly. Software can fail silently, so you decide in advance which numbers to watch.

13

Why it matters

Most outages start with a change. Agents make changes often, so release habits matter even more.

Settings drift. The code is fine; the real environment is missing one line of configuration.

Data doesn't rewind. Thousands of new bookings written in a new shape don't go back on their own.

Quiet isn't healthy. If nobody watches the checkout success rate, 9% of payments failing looks like a calm Friday evening.

Bills don't wait. A stuck loop calling a paid AI service runs all weekend, and can use up a limit other features need.

14

Check your understanding

Q1/3Scenario
Put the release of an agent-built checkout feature in order.
  1. 1.Ramp to 25%, then 100%, checking between steps
  2. 2.CI green, smoke test on staging, environment diff clean
  3. 3.Deploy plan in the PR: settings, migration, rollback, metrics
  4. 4.Deploy to production with the flag off
  5. 5.Flag on for 5%; compare checkout success with baseline
How confident are you?
15

Practice ladder

Step 1 · Trace

Trace Friday, minute by minute

Debugging Warm-up

Trace Friday's timeline. For each moment, say what was happening in the system and which single measure would have stopped or shortened it.

  • 17:30 deploy, no flag, all guests
  • 17:34 checkout errors 9%
  • 17:41 Omar notices via support
  • 17:50 rollback: errors 30%
  • 18:20 fixed forward
Step 2 · Complete

Complete the rollback plan

Scenario Practice

Complete the rollback section of the Tuesday redeploy plan.

  • Fastest undo: ___ (time: ___)
  • Second option: ___ (time: ~4 min). Safe because ___.
  • Data written by the new version: ___ (what happens to it after an undo?)
  • Who can do it: ___
  • Rehearsed: ___
Step 3 · Solve

Deploy plan for the new search ranking

Design Practice

Maya had Claude Code build a new hotel search ranking that uses review scores and recent bookings. It adds a nightly job that computes scores with an AI model, a new table hotel_scores, and a setting RANKING_MODEL_KEY. Search drives most bookings.

Write the deploy plan: what it needs, how it's switched on, what proves it works (with stop rules), how to undo it, and cost guardrails.

Step 4 · Decide

Drop the legacy column today?

Decision Stretch

Thursday 16:00. Claude Code opens a cleanup PR: it removes all code using bookings.legacy_status (replaced by status three months ago) and drops the column in the same migration. Tests are green. Karim wants the cleanup done before the quarter ends tomorrow.

Write your decision: "We chose X over Y because…, accepting the risk of…", with what you'd ask for.

16

Explain it yourself

Karim asks: "The tests passed and Claude deployed it. Why did checkout break, and what changes next time?" Answer him.

0 words