Episode 91 · Green CI, wrong prices

The testing pyramid: which test catches which bug?

Thursday, 10:30: a one-line pricing change passed CI yesterday, and 214 Saturday-night stays have since sold at weekday prices. StayHub's 250 browser tests take 70 minutes, fail randomly one run in five, and not one of them checks a Saturday price.

Different tests catch different bugs. Here's how to pick the cheapest test that can still catch each one.

30 min intermediateTesting pyramid: balancing unit, integration, contract and e2eChoosing the cheapest test level that catches a bugFlaky tests and keeping test results trustworthyRunning tests in CI: fast gates firstAPI as a contractPipeline stages & fast feedback

Problem
StayHub ships changes to booking, pricing and payments every day. Bugs reaching production cost refunds and trust; a slow, unreliable test suite costs developer time and slows every release.
Why the simple system fails
Testing only by clicking through the app (manually or with lots of browser tests) is slow, flaky, and doesn't tell you which line broke. Testing only with mocked unit tests misses everything that happens between components: SQL, migrations, API shapes, layout.
Engineering concept
Each level of test sees a different class of failure. Put many fast, precise tests at the bottom, enough integration and contract tests to cover the seams, and a thin layer of end-to-end tests for critical journeys. Push every check down to the cheapest level that can see the bug.
Technology
Unit test runners (Vitest, Jest, pytest), integration tests against real databases in containers, consumer-driven contract tests, browser e2e tools (Playwright, Cypress), CI pipelines with staged gates, flaky-test quarantine.
Trade-offs
Lower levels are fast and precise but can pass while the system is broken; higher levels give realistic confidence but are slow, costly and flaky. Mocks speed tests up but can drift from reality. The right shape depends on where your bugs actually come from.

The problem

2 min

Thursday, 10:30. The pricing change was merged yesterday at 15:00.

Maya · Product manager
214 Saturday-night stays sold at weekday prices since yesterday. About €9,600 below the hotels' rates.
Omar · On-call SRE
CI was green. 250 browser tests, 70 minutes. And one run in five fails at random anyway.
You · Engineer at StayHub
So we need a browser test that books a Saturday?
Lina · Senior engineer, your mentor
Maybe. Guess first.

The big idea

4 min

250 tests, and none of them looked

The big idea

For each kind of bug, put the check in the cheapest test that can see it.

Inspecting a hotel, you check every door lock at the factory, check that installed locks talk to the key-card system, and send one mystery guest through check-in to check-out. You'd never test door locks by sending a mystery guest to every room.

How it works

5 min

One kind of bug, one level

1
Rules → unit tests
A checks one function with no network or database. Milliseconds; failures point at the line. Weekend rates, discounts, rounding: hundreds of cases in seconds.
2
Data → integration tests
An runs your code with a real dependency, usually Postgres in a container. Seconds. SQL, migrations, constraints, two guests booking the last room at once.
3
Between services → contract tests
Booking renames a field the email service reads. Checked in each service's own build.
4
Journeys → a few e2e tests
An drives the deployed app through a browser. Realistic, slow, fragile. Keep about 15 journeys (search, book, pay on a phone), not 250 screens.
5
The shape: a pyramid
The : many unit tests, fewer integration and contract tests, a thin e2e layer. StayHub has an ice-cream cone: upside down, slow and , so nobody reads its failures.
6
Why flaky suites stop working
250 tests at 0.5% flake each: 0.995²⁵⁰ ≈ 29% of runs fully green. Red becomes normal, so a real red is ignored too.

Watch a decision being made

4 min

Worked example

Lina reshapes the suite

Stop the next pricing bug from shipping, and turn a 70-minute, flaky, ignored suite into one that answers in about 8 minutes and that people trust.

1Reproduce the bug in a unit test

Write the failing test first, at the cheapest level that sees it.

it("charges the weekend rate on Friday and Saturday nights", () => {
  const p = priceFor({ weekday: 90, weekend: 135, checkIn: "2026-10-09", nights: 2 }); // Fri + Sat
  expect(p.total).toBe(270);
});

Why: Red before the fix proves the test can see the bug; green after proves the fix.

2Ask what each e2e test guards
For each of the 250 browser tests: what bug would make it fail? A pricing total, a form rule, a query, a full journey?
Why: Most browser tests really check one rule or one query, through a slow, fragile path.
3Move each check down
Rule checks become unit tests, data checks integration tests on real Postgres. Only 15 full journeys stay e2e. Target: ~800 unit, ~150 integration, ~40 contract.
Why: Same behaviour covered, a fraction of the time, and failures that say what broke.
4Cheap gates first, flakiness is a bug
Lint and types, then unit, then integration and contract, then journeys; stop at the first failure. No blanket retries: quarantine a flaky test with an owner and a deadline.
Why: Most failures show in 2 minutes, and a suite that's rarely red for no reason gets read.

In real life

3 min

Pyramid vs honeycomb

Google: the classic pyramid

Its testing blog published "Just Say No to More End-to-End Tests" in 2015
Suggested roughly 70% unit, 20% integration, 10% e2e as a starting split

Fast, precise failures where logic is the risk.

Spotify: the testing honeycomb

Its engineering blog described a "honeycomb" for microservices in 2018
Mostly integration tests of each service through its real API and dependencies

Realistic confidence where services are mostly glue.

What goes wrong

3 min

Blanket retries for flaky tests

Real races get retried into production, and red stops meaning anything.

Instead

Quarantine with an owner and a deadline; fix the cause.

Mocking the database in every test

SQL, migrations and constraints never run; green suite, failing production.

Instead

Fakes for logic; a real database for data access code.

Chasing 100% line coverage

Coverage counts lines run, not behaviour checked.

Instead

Use it to find untested risky code; judge the suite by bugs that escape.

Trade-offs

3 min

Level	Speed per test	Catches	Blind to
Unit	~1–20 ms	Logic, edge cases, rounding	SQL, wiring, layout
Integration	~0.5–2 s	Queries, migrations, transactions	Other teams' services, UI
End-to-end	~30–60 s	Critical journeys, layout, config	Where exactly it broke

Friday, 11:00. The plan to move 235 browser tests.

Maya · Product manager
Those 250 tests do what guests do. Deleting 235 feels like turning off the smoke alarms.
Lina · Senior engineer, your mentor
We're moving each check to a test that runs in milliseconds and doesn't fail at random.
Maya · Product manager
A unit test won't notice if the "Book" button disappears on phones.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which behaviours must be pinned before any change ships: the Saturday price, the last room sold once, refund limits. You name them; the agent writes them.
At which level each check lives: rules as , the race as an on real Postgres, journeys as ~15 .
What happens to a : quarantine with an owner, fix the cause. Never blanket retries.

Tell Claude Code

A one-line pricing change priced Saturday nights at the weekday rate and CI stayed green. Before fixing it, write unit tests for priceFor pinning the rules: weekday, Friday and Saturday nights, a stay crossing a weekend, member discount rounding. Plain inputs, expected euro amounts, no mocks. Run them and show me which fail; I expect the Saturday ones to. Then fix the bug in a separate commit and rerun. Don't change other tests or add retries.

Check before you accept

Read the test names and amounts like a spec: would a hotel manager agree with each price?
You saw the Saturday tests fail before the fix. A test that never failed may test nothing.
No existing expectation was edited to make it pass without your approval.

Red flags

It changes the expected value to whatever the code currently returns.
It adds retry: 3 or skips a test to get CI green.
It reports "all tests pass" without saying how many ran.

Check yourself

2 min

Q1/3Scenario

Broken code: which test catches it most cheaply? Match each bug to the level.

Tax computed on the discounted price instead of the full price

New migration adds a NOT NULL column with no default to a populated table

Booking service renames checkIn to check_in; email service still reads checkIn

After a CSS change, the Pay button is hidden behind the cookie banner on mobile

How confident are you?

Next episode: the booking tests mock the payment provider, every one passes, and the first real refund fails in production.

Finished?

Mark it complete to update your skill map.

Next: Mocks, fakes and fixtures: testing without lying to yourself

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Why a flaky suite stops working

Worked example

Lina reshapes the suite

Stop the next pricing bug from shipping, and turn a 70-minute, flaky, ignored suite into one that answers in about 8 minutes and that people trust.

1Reproduce the bug in a unit test
Write the failing test first, at the cheapest level that sees it.
ts
```
it("charges the weekend rate on Friday and Saturday nights", () => {
  const p = priceFor({ weekday: 90, weekend: 135, checkIn: "2026-10-09", nights: 2 }); // Fri + Sat
  expect(p.total).toBe(270);
});
```
Why: Red before the fix proves the test can see the bug; green after proves the fix. It then guards against the regression forever.
2Ask what each e2e test guards
Go through the 250 browser tests. For each: what bug would make it fail? Pricing totals, form validation, a query's results, a full journey?
Why: Most browser tests are really checking one rule or one query, through a slow, fragile path.
3Move each check down
Rule checks become unit tests (~200 of them). Data checks become integration tests on a real database. Only full journeys stay e2e: 15 of them.
Why: Same coverage of behaviour, a fraction of the time, and failures that say exactly what broke.

4Test data rules against a real database

Run booking and inventory tests against a disposable Postgres container, including concurrency.

it("never sells the last room twice", async () => {
  await seedRoom({ id: "r1", date: "2026-12-24", available: 1 });
  const attempts = await Promise.allSettled([
    bookRoom(db, { roomId: "r1", guestId: "g1", date: "2026-12-24" }),
    bookRoom(db, { roomId: "r1", guestId: "g2", date: "2026-12-24" }),
  ]);
  expect(attempts.filter((a) => a.status === "fulfilled")).toHaveLength(1);
});

Why: A fake in-memory repository has no transactions or locks. Only a real database can show a race or a broken migration.

5Run cheap gates first
Lint and types, then unit, then integration and contract in parallel, then the 15 journeys. Stop at the first failure.
Why: Most failures show up in 2 minutes; nobody waits 70 minutes to learn about a typo.
6Treat flakiness as a bug
Remove blanket retries. Track each test's pass rate on the same commit; quarantine a flaky test with an owner and a deadline, and fix the cause.
Why: With 250 tests at 0.5% flake, only 29% of runs are green, so people stop reading red. A trusted suite is rarely red for no reason.

Level	Speed per test	Typical count	Catches	Blind to
Unit	~1–20 ms	Hundreds to thousands	Logic, edge cases, rounding, validation	SQL, wiring, other services, layout
Integration	~0.5–2 s	Tens to hundreds	Queries, migrations, constraints, transactions	Other teams' services, UI
Contract	~0.1–1 s	Per API/event pair	Renamed or removed fields between services	Business logic behind the API
End-to-end	~30–60 s	A handful to a few dozen	Critical journeys, layout, config	Where exactly it broke; rare edge cases

Orders of magnitude; measure your own suite.

Interactive simulation

Visual diagram

Browser (e2e)

Booking service

Pricing module (unit)

PostgreSQL (integration)

Email service

Payment provider (sandbox)

Click a component

What each test level exercises in StayHub's booking flow. Wider scope = more realism, more cost, vaguer failures.

Analogy

Building a hotelTesting software

Building a hotelChecking each door lock in the factoryTesting softwareUnit tests
Building a hotelLock talks to the key-card systemTesting softwareIntegration tests
Building a hotelElectrician's sockets match the furniture plugsTesting softwareContract tests
Building a hotelA mystery guest checks in and outTesting softwareEnd-to-end tests

You'd never test every door lock by sending a mystery guest to every room. Where the analogy breaks: software tests are free to rerun thousands of times a day, so speed and reliability matter even more than in a building.

Check your understanding

Q1/3Reasoning

A suite has 100 e2e tests, each independently flaky with a 1% chance of a false failure per run. Roughly what percentage of runs will fail at least once for no real reason?

How confident are you?

Practice ladder

Step 1 · Trace

Which test catches each bug?

Scenario Warm-up

For each StayHub bug, trace which test level is the cheapest that would catch it, and why the levels below it can't.

A 10% member discount on 3 nights at €33.33 gives €90.00 (each night rounded first) instead of €89.99.
A migration adds a NOT NULL column with no default; it fails on the production table, which has rows.
The booking service renames guestName to guest_name in the event the email service reads.
On phones, the cookie banner covers the "Book" button.
Two guests book the last room at the same instant and both succeed.

Step 2 · Complete

Order the CI pipeline

Design Practice

Lina's CI plan has two gaps. Fill in step 2 and the rule in step 6.

Lint and type-check (~1 min)
___
Integration tests on a Postgres container (~4 min), in parallel with
Contract tests (~1 min)
15 e2e journeys on a preview environment (~3 min)
Rule for a test that fails, then passes on a retry of the same commit: ___

Step 3 · Solve

Test the new cancellation fee

Design Practice

Maya's new feature: cancelling within 48 h of check-in costs one night's price; earlier cancellations are free; non-refundable rates refund nothing. The booking service stores the fee in a new cancellation_fee column, emits a BookingCancelled event that the email and payments services read, and shows the fee on the "Cancel" screen.

Design the tests: which checks at which level, roughly how many, and what stays e2e.

Step 4 · Decide

A 100% coverage target?

Decision Stretch

After the pricing bug, Karim proposes: "From now on, every pull request must keep line coverage at 100%." Today's coverage is 64%. The pricing module, where the bug was, was at 91% coverage: the weekend branch ran in a test, but nothing checked its result.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain the testing pyramid to a teammate who wants to add 50 more browser tests, and how you'd decide where a new test should go.

0 words