Episode 57 · It worked on my laptop

CI/CD pipelines

Tuesday, 15:40: a change that passed every test on Lina's laptop and in CI reaches production, and twenty minutes after the merge, checkout fails for every guest.

An automatic assembly line that checks every change and carries it safely to real users, so releasing becomes boring.

30 min intermediateCI vs continuous delivery vs continuous deploymentPipeline stages & fast feedbackBuild once, promote the same artifactDev, staging & production environmentsImages vs containers

Problem
A booking team wants to ship fixes and features several times a day without breaking checkout.
Why the simple system fails
Manual builds on a laptop and hand-run deploys are slow, inconsistent and scary, so teams release rarely, in big risky batches, and bugs are found by customers.
Engineering concept
Automate the path from commit to production: integrate and test every change (CI), build one immutable artifact, and promote it through environments with automated gates (CD).
Technology
Pipeline-as-code (GitHub Actions, GitLab CI and similar), container images in a registry, staging environments, smoke tests, progressive rollouts.
Trade-offs
Every stage catches some bugs and costs minutes. Too few stages let bugs escape; too many make feedback slow and push people to bypass the pipeline.

The problem

2 min

Tuesday, 15:40. Twenty minutes after Lina merged a small price-display change.

Omar · On-call SRE
Checkout: 100% errors. TypeError inside money-fmt. About 17 bookings a minute lost.
Lina · Senior engineer, your mentor
My PR. Tests passed on my laptop and in CI.
Omar · On-call SRE
Production has money-fmt 3.0.0; the lockfile pins 2.4.1. The deploy script runs pnpm install and builds on each prod server.

The big idea

4 min

Same code, different build

The big idea

Build once, then deploy that same artifact everywhere.

Rolled back at 16:06: 26 minutes, about 430 bookings lost. "Add more tests" wouldn't help: more tests on one build say nothing about a different build in production.

How it works

5 min

From push to production

1
Fast checks first
builds and tests every change automatically before it joins main. Lint and types first (1.5 min): silly mistakes surface while you remember the change.
2
Tests in parallel
Unit tests and integration tests against a real PostgreSQL, side by side, plus a dependency scan. About 9 minutes to an answer.
3
Build once
A is the packaged result, built once per commit with a frozen lockfile: booking-api-<commit>.tgz, pushed to the artifact store, never rebuilt.
4
Staging
mirrors production with fake data; only settings differ. The same bundle runs there, plus smoke tests: search, book, pay, cancel.
5
Production, carefully
The same bundle to 5% of traffic for 10 minutes, rolling back by itself if errors pass 1%. Merge to production ≈ 40 minutes.
6
Who presses the last button
: a person promotes the tested bundle with one click. : the pipeline does it when the canary is healthy.

Watch a decision being made

4 min

Worked example

Lina rebuilds the pipeline

What's tested must be exactly what ships; PR feedback under 10 minutes; Tuesday's kind of bug unable to reach every guest.

1Order checks by speed
Lint + types (1.5 min); then unit (3 min) and integration vs real Postgres (4 min) in parallel. About 9 minutes.
Why: The cheapest checks catch most mistakes; under 10 minutes, people wait instead of batching changes.

2Build once

On merge: one build with a frozen lockfile, named with the commit SHA. No later job builds anything.

yaml

build:
  - run: pnpm install --frozen-lockfile && pnpm build
  - run: tar czf booking-api-${{ github.sha }}.tgz dist node_modules
  - run: ./upload-artifact.sh booking-api-${{ github.sha }}.tgz
deploy-staging:
  needs: build
  - run: ./deploy.sh staging booking-api-${{ github.sha }}.tgz

Why: This is the fix for Tuesday: every later stage tests what will ship.

3Staging, then a canary
Same bundle to staging with smoke tests; then 5% of production for 10 minutes, auto-rollback above 1% errors or doubled p95.
Why: Tuesday's bug now fails in staging; anything else is limited to 5% for about a minute.
4Ban side doors
No git pull, installs or builds on production servers. Rollback = deploy the previous bundle.
Why: A manual fix that bypasses the pipeline gets silently undone by the next deploy.

In real life

3 min

Deploy all day versus release trains

Etsy: continuous deployment

Has publicly described deploying its website many times a day
Small changes, heavy monitoring, quick fixes forward
Works because there's one live version

Fits a website where changes can go live in minutes.

Chrome: release channels

Publicly ships through Canary, Dev, Beta and Stable, with a stable release every few weeks
Integrated continuously, released on a schedule
Installed software can't be rolled back instantly

Fits installed software where a bad release is hard to undo.

What goes wrong

3 min

Ignoring [[flaky-test|flaky tests]]

People re-run until green, so real failures get ignored too.

Instead

Quarantine the same day, with an owner and a deadline; treat "failed then passed" as a failure.

Rebuilding per environment

Production runs something staging never tested.

Instead

Build once, tag with the commit, promote that artifact.

A 60-minute pipeline

People batch changes and start bypassing it.

Instead

Fast-to-slow ordering, parallel jobs, caching.

Trade-offs

3 min

Approach	Fits	You pay
Continuous delivery (manual promote)	Regulated domains, weaker tests	Waiting; changes pile up; rubber-stamping
Continuous deployment (automatic)	Mature teams with good tests and monitoring	Excellent canaries, rollback and flags
More pipeline stages	A class of bug keeps escaping	Slower feedback; more flakiness

Wednesday, 10:00. After the outage.

Karim · CTO
From now on, QA approves every production deploy by hand.
Omar · On-call SRE
At 60 merges a day, that's a two-hour queue. People will batch changes, and batches make outages big.
Karim · CTO
A canary is a machine guessing. I want a human.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which checks must be green before merging, and that nobody, human or agent, may skip them.
The build-once rule: one bundle per commit SHA, promoted, never rebuilt.
What to do with a flaky test: quarantine and fix it on a deadline, never "just re-run".

Tell Claude Code

Our deploy script runs git pull, pnpm install (without --frozen-lockfile) and pnpm build on each production VM; that's how money-fmt 3.0.0 reached production untested. Change the pipeline: frozen lockfile on every install; booking-api built once per commit on main into booking-api-<sha>.tgz in our artifact store; staging and production deploy that bundle; production never builds. Keep lint and test stages. Plan first: changed files and stage order. Then prove on a test commit that staging and production got the same file (name and checksum).

Check before you accept

Exactly one build step; later stages only fetch and deploy the bundle.
--frozen-lockfile on every install.
No required check removed, made optional or set to "continue on error".

Red flags

It makes a failing test pass by skipping it or adding automatic retries.
It names bundles latest, or rebuilds in the production job.
It removes the canary or smoke test "to speed things up".

Check yourself

2 min

Q1/3Reasoning

Every merge to main is built, tested and deployed to staging automatically. A product owner clicks 'Release' to send it to production when ready. What is this?

How confident are you?

Next episode: someone fixes a production setting by hand in the cloud console, and the next infrastructure run quietly puts the old value back.

Finished?

Mark it complete to update your skill map.

Next: Infrastructure as code

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

A pipeline that answers in nine minutes

Worked example

Lina rebuilds the pipeline

Make sure what's tested is exactly what ships, give PR feedback in under 10 minutes, and get a merged change to production within an hour, with Tuesday's kind of bug unable to reach every guest.

1Order checks by speed
Lint + types (1.5 min) first; then, once dependencies are installed and a real PostgreSQL container is up (about 3 min), unit tests (3 min) and integration tests (4 min) run in parallel; dependency and secret scan alongside. About 9 minutes.
Why: Most mistakes are caught by the cheapest checks. Running slow suites in parallel keeps the answer under 10 minutes, so people wait for it instead of batching changes.
2Build once
On merge to main: one build with --frozen-lockfile, bundle booking-api-<commit sha>.tgz, push to the artifact store. No later job builds anything.
Why: This is the fix for Tuesday. Every later stage now tests the artifact that will ship.
3Deploy to a staging that mirrors production
Same bundle, same infrastructure code, smaller size, synthetic data, a fake payment provider. Smoke tests: search, book, pay, cancel.
Why: Staging catches what tests can't see: missing environment variables, broken deploy config, journeys across services.
4Release carefully
Production: the same bundle to 5% of traffic for 10 minutes; automatic rollback if the error rate is over 1% or p95 latency doubles; then 100%.
Why: Some bugs only appear with real traffic and data. A canary keeps them small and short.
5Ban side doors
No git pull, pnpm install or builds on production servers; production changes only through the pipeline. Rollback = deploy the previous bundle.
Why: A manual fix that bypasses the pipeline gets silently undone by the next deploy, and nobody can tell what's running.

Environment	Purpose	Data	Who deploys
Preview (per PR)	Try changes fast	Fake or seeded	Pipeline, per PR
Staging	Validate the release in a production-like setup	Synthetic or anonymised, never raw personal data	Pipeline, on every merge
Production	Real guests	Real	Pipeline only

Interactive simulation

Visual diagram

Git (main)

CI: lint, test, scan

Build bundle

Artifact store

Staging

Production

Monitoring

Click a component

From a merged commit to production, with one artifact promoted through environments.

Analogy

In a restaurant kitchenIn a pipeline

In a restaurant kitchenThe cook tasting each sauce as it's madeIn a pipelineCI: small checks on every change
In a restaurant kitchenThe finished, plated dishIn a pipelineThe build package, never rebuilt
In a restaurant kitchenThe head chef inspecting at the passIn a pipelineStaging
In a restaurant kitchenPlates waiting, ready to serveIn a pipelineContinuous delivery
In a restaurant kitchenA conveyor belt straight to tablesIn a pipelineContinuous deployment
In a restaurant kitchenServing one table first and watchingIn a pipelineA canary release

Where the analogy breaks: a kitchen cooks a fresh plate for each guest. A pipeline builds the package once, and the very same copy is served to every user.

Check your understanding

Q1/3Reasoning

Order these stages for the fastest useful feedback:

1.Lint and typecheck
2.Unit tests
3.Deploy to staging + smoke tests
4.Canary deploy to production
5.Build and push the image
6.Integration tests with a real database

How confident are you?

Practice ladder

Step 1 · Trace

Time a commit through two pipelines

Estimation Warm-up

Pipeline A runs stages one after another: lint 1 min, unit 4 min, integration 10 min, build 5 min, staging deploy 4 min, E2E 15 min, manual approval 120 min, production 5 min.

Pipeline B runs lint, unit and integration in parallel (the slowest decides), then build, staging + smoke tests (6 min in total), then a 10-minute canary to production with automatic promotion.

Push to production time for a clean commit in A and in B.
In each pipeline, at what minute does a developer learn that an integration test failed?

Step 2 · Complete

Fix the pipeline file

Code Practice

This pipeline file still has Tuesday's problem. Fill in the two gaps so production runs the exact bundle staging tested, and explain why your production step must not build anything.

build:
  needs: test
  steps:
    - run: pnpm install --frozen-lockfile && pnpm build
    - run: tar czf booking-api-${{ github.sha }}.tgz dist node_modules
    - run: ./upload-artifact.sh booking-api-${{ github.sha }}.tgz

deploy-staging:
  needs: build
  steps:
    - run: ./deploy.sh staging ___        # gap A
    - run: ./smoke-test.sh https://staging.stayhub.com

deploy-prod:
  needs: deploy-staging
  steps:
    - run: ___                            # gap B
    - run: ./canary.sh --percent 5 --minutes 10 --max-error-rate 0.01

Step 3 · Solve

A pipeline for the partner API

Design Practice

Design the CI/CD pipeline for the partner API (TypeScript, PostgreSQL, VMs set up by scripts). Requirements:

PR feedback in under 10 minutes.
Production within 1 hour of merge.
Partner booking and cancellation must never break silently.
Database schema changes happen about twice a week.
300 partner agencies depend on the API's responses not changing shape.

List the stages in order, what each catches, and how a bad release is detected and undone.

Step 4 · Decide

Add an end-to-end stage?

Decision Stretch

Two checkout bugs reached production last month (the Pay button hidden on small phones; a broken coupon field). Maya asks for end-to-end (E2E) browser tests.

Options: - A: a full E2E suite (120 tests, 30 minutes) on every PR. - B: 8 critical-journey E2E tests (search → book → pay, cancel, coupon, mobile viewport) run on staging after every merge, about 10 minutes, blocking promotion if they fail.

PR feedback is 9 minutes today; the team won't accept 40. Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain what happens between a developer merging a change and guests using it, and why each step exists.

0 words