Metrics and experiments: knowing whether a change actually helped
Monday, 10:00. A week after the new one-page checkout launched in the Android app in Spain and Portugal, Maya's dashboard says conversion rose from 4.0% to 4.3%. Omar's says 7-day cancellations rose from 6.0% to 7.5%. It was also a holiday weekend, with a campaign running.
Shipping a change isn't the goal. Here's how to tell whether it actually helped, or just looked like it did.
- Problem
The booking team redesigned checkout. Conversion went from 4.0% to 4.3% the week after launch. Everyone celebrates.
- Why the simple system fails
That week also had a holiday weekend and a marketing campaign. Nobody can say whether the redesign helped, did nothing, or hurt and was masked by the campaign. Meanwhile cancellations quietly rose.
- Engineering concept
Define the outcome you care about (north star), the levers that move it (input metrics) and what must not get worse (guardrails); then compare randomised groups at the same time, with a sample size fixed in advance.
- Technology
Event tracking, feature flags with sticky bucketing, A/B testing analysis (two-proportion tests, confidence intervals), experiment dashboards, RICE scoring.
- Trade-offs
Experiments give causal answers but need traffic, time and discipline; small products often can't detect small effects and must rely on bigger bets and qualitative evidence.
The problem
2 min- Maya · Product managerConversion went from 4.0% to 4.3% in launch week. It worked!
- Omar · On-call SRE7-day cancellations went from 6.0% to 7.5%. Same week.
- Lina · Senior engineer, your mentorYou lead. What else happened that week?
The big idea
4 minCompare random groups at the same time, with the metrics and sample size fixed in advance.
"I took the pill and felt better" is a testimonial. A trial gives the pill to a random half at the same time; whatever else happens that month happens to both.
How it works
5 minAn randomly splits users into groups that see different versions at the same time. You grow a (bookings) while must not get worse.
- 1Metrics first
Primary: booking conversion per checkout visitor (4.0%). Guardrails: 7-day cancellations, payment errors, support contacts, p95 latency.
- 2Size it
The is the smallest change worth finding: +0.4 pp. That needs ~38,400 guests per variant: 3 full weeks.
- 3Assign at random, stickily
Hash guest + experiment ID so a guest always sees the same version; log exposure when they actually see it.
- 4Check the plumbing
Is the split really 50/50? A means a bug, not a result.
- 5Analyse once
After 21 days: the difference, its (the plausible range), the , and every guardrail.
Watch a decision being made
4 minYou run the checkout test properly (Lina checks)
Find out what the checkout itself does to bookings and cancellations, with an answer you can defend.
- 1Write metrics before data
Primary metric and four guardrails, written down.
Why: Lina: "Before anyone opens a dashboard." Choosing later lets you find whichever looks good.
- 2Fix sample size and duration
Before any data:
textn per variant ≈ 16 × p × (1 − p) / δ² = 16 × 0.04 × 0.96 / 0.004² ≈ 38,400 ~2,000 per arm per day → ~19 days → 3 full weeksWhy: Full weeks include every weekday; a fixed end stops you stopping at a lucky moment.
- 3Don't stop on day 5
B shows p = 0.03 on day 5; Maya wants to ship. Keep running.
Why: Early peeks are noisy; stopping at the first good p multiplies false wins.
- 4Analyse once, then decide
Day 21: A 1,680/42,000 (4.0%), B 1,848/42,000 (4.4%). CI +0.13 to +0.67 pp, p ≈ 0.004. Guardrails flat.
Why: Even the low end of the range is a real gain, and cancellations are the same in both arms.
In real life
3 minBooking.com: experiment on almost everything
- Widely described as running very many A/B tests at once
- Huge traffic makes small effects detectable
Fits high-traffic, reversible changes.
Apple: design judgement for big launches
- Known for decisions by small design teams
- A new device can't be split 50/50
Fits big, irreversible bets.
What goes wrong
3 minUrgency messages can raise bookings and raise cancellations more.
Pre-register guardrails and require them to hold.
Huge traffic makes tiny effects significant.
Read the interval against the effect that would matter.
Test 20 segments and one looks significant by chance.
Pre-register segments; treat later wins as new hypotheses.
Trade-offs
3 min| Method | Fits when | Costs |
|---|---|---|
| A/B test | High traffic, reversible, detectable effect | Weeks; flags and analysis |
| Ship and monitor | Low traffic, big or required changes | Confounded by season and campaigns |
| Qualitative research | Early ideas; explaining why | Doesn't measure how much |
- Maya · Product managerLet's A/B test the payout page too. Primary metric: payout tickets per hotel.
- Karim · CTOWe have about 10,000 hotels. Detecting a 10% drop needs about 9,000 per arm.
- Karim · CTOShip it behind a flag, watch tickets, run five usability sessions.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The primary metric and guardrails, before any data.
- The MDE (+0.4 pp), which fixes sample size and duration, and the rule that nobody stops early.
- The randomisation unit (the user) and when exposure is logged (when the variant is seen).
Set up the one-page checkout as an A/B test, id checkout_onepage. Plan first. Android users in Spain and Portugal only, 50/50 sticky by user id via `src/flags/`; log exposure when the page renders the variant; tag booking, cancellation, payment-error and support events. Write `analysis/checkout_onepage.sql`: check the sample ratio first and refuse to report if off, then conversion and four guardrails per arm with 95% intervals. 21 days, ~42,000 per arm. No p-value dashboards before day 21.
- The same user gets the same variant on every visit and server.
- A deliberate 51/49 split on 100k test users stops the analysis.
- Every guardrail appears with its own interval.
- A live daily p-value with "stop when significant".
- Randomisation by session, so one guest sees both checkouts.
- "97% chance B is better" from p = 0.03.
Check yourself
2 minNext episode: the booking path still runs in a single region. Karim wants a second region by summer, Omar wants fewer moving parts, and both have a point.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You run the checkout test properly (Lina checks)
The one-page checkout went to everyone during a holiday week, and two dashboards disagree. Find out what the checkout itself does to bookings and cancellations, with an answer you can defend.
- 1Write the hypothesis and metrics
Hypothesis: a one-page checkout reduces drop-off in the Android app, which converts at half our ~8% average (pilot: Spain and Portugal, ~4,000 checkout visitors a day). Primary: booking conversion per checkout visitor (baseline 4.0%). Guardrails: 7-day cancellations, payment errors, support contacts per booking, p95 latency.
Why: Choosing metrics after seeing data lets you find whichever one looks good. Lina: "Write them down before anyone opens a dashboard."
- 2Fix sample size and duration
MDE +0.4 pp (10% relative): n ≈ 16 × 0.04 × 0.96 / 0.004² ≈ 38,400 per variant. ~2,000 per arm per day → ~19 days → 3 full weeks.
Why: Full weeks include every weekday. A fixed end date stops you from stopping at a lucky moment.
- 3Assign stickily, log exposure
Hash experiment ID + guest ID into a bucket; log
experiment_exposurewhen the checkout renders, not when the guest is assigned.tsconst h = createHash("sha256").update(`${experimentId}:${userId}`).digest(); const variant = h.readUInt32BE(0) % 100 < 50 ? "B" : "A"; analytics.track("experiment_exposure", { experimentId, variant, userId });Why: The same guest must always see the same version, and only guests who saw a checkout belong in the comparison.
- 4Check the plumbing first
Before reading results: is the split close to 50/50? Are bots filtered equally? Does B win on every metric at once (suspicious)?
Why: A sample ratio mismatch or a too-good result usually means a bug in assignment or logging, and then the result means nothing.
- 5Analyse once, then decide
Day 21: A 1,680/42,000 (4.0%), B 1,848/42,000 (4.4%). Lift +0.4 pp, 95% CI ≈ +0.13 to +0.67 pp, p ≈ 0.004. Guardrails flat. Ship B, keep watching cancellations for a month.
Why: The interval excludes zero and includes the effect we said would matter. Cancellations lag, so the guardrail watch continues after the decision.
from math import sqrt
from statistics import NormalDist
def two_proportion_test(conv_a, n_a, conv_b, n_b):
p_a, p_b = conv_a / n_a, conv_b / n_b
pooled = (conv_a + conv_b) / (n_a + n_b)
z = (p_b - p_a) / sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
p_value = 2 * (1 - NormalDist().cdf(abs(z)))
se = sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
return z, p_value, (p_b - p_a - 1.96 * se, p_b - p_a + 1.96 * se)
print(two_proportion_test(1680, 42_000, 1848, 42_000))
# z≈2.89 p≈0.004 CI≈(+0.0013, +0.0067)Visual diagram
The machinery of an A/B test: assignment, exposure logging, events, analysis.
Analogy
- A medical trialThe patient's long-term healthA product experimentThe north star metric
- A medical trialSide effects to watchA product experimentGuardrail metrics
- A medical trialDrug and placebo, similar people, same timeA product experimentRandom A/B groups at once
- A medical trialTrial size fixed in advanceA product experimentSample size decided before starting
- A medical trialStopping on the first good dayA product experimentPeeking: how false wins happen
Where the analogy breaks: medical trials are slow and strictly supervised. Product tests are cheap and fast, which makes it very tempting to peek, stop early, and announce a "cure" that was just luck.
Check your understanding
Practice ladder
Read an experiment result
Scenario Warm-upA test of a new hotel-page photo gallery ran for its planned 14 days, 50/50.
| Users | Bookings | 7-day cancellations | |
|---|---|---|---|
| A (old) | 50,600 | 2,530 | 152 |
| B (new) | 48,900 | 2,543 | 153 |
- Compute conversion for each arm.
- Check the split. Is anything suspicious?
- Would you trust the result yet?
Size the date-change test
Estimation PracticeLina's plan for testing a new self-service date-change flow has gaps. Fill them in.
- Primary metric: share of guests with an upcoming stay who change dates without contacting support. Baseline p = 10%.
- Smallest effect worth detecting: +2 pp (to 12%).
- n per variant ≈ 16 × p × (1 − p) / δ² = ___
- Eligible guests per day: 1,500, split 50/50 → ___ per arm per day → ___ days → run for ___.
- Guardrails: ___ (name at least two).
The "only 1 room left" banner
Design PracticeMaya wants to test a red banner on hotel pages: "Only 1 room left at this price!" (shown when it's true). She expects more bookings.
Design the experiment: hypothesis, primary metric, guardrails, sample size reasoning (baseline conversion on hotel pages is 3%, she hopes for +10% relative), duration, and what result would make you ship, change or drop it.
Ship it with a wobbly guardrail?
Decision StretchA 3-week test of "book now, pay later" for stays over 7 nights finished:
- Conversion: +0.5 pp, 95% CI +0.2 to +0.8 pp (baseline 4.0%).
- 7-day cancellations: +0.3 pp, 95% CI −0.1 to +0.7 pp (baseline 6.0%), not significant.
- Each extra booking is worth ~45 € in commission; each cancellation costs ~12 € in handling and lost inventory.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Your team says the new checkout "worked" because conversion went up the week after launch. Explain what you'd do to actually know.