Skip to content
Episode 93 · A rename and a Black Friday

Contract, load and security tests: the bugs that live between and under

Monday, 9:00: the booking team will rename guestName on Wednesday, and two other teams read that field. Black Friday is three weeks away, with 4× the usual search traffic. Every test passes, and nobody knows what breaks first.

Some bugs hide between services, some only appear under heavy traffic, some only under attack. Normal tests miss all three.

30 min intermediateConsumer-driven contract testing between servicesBackward and forward schema compatibilityLoad, stress, spike and soak testsLoad testing to find the breaking pointSecurity testing: SAST, DAST, dependency scans, fuzzingp50 / p95 / p99 latency
  1. Problem

    Each service's tests are green, yet production breaks: a renamed field between two services, a search that dies at 300 concurrent users, a review form that runs attackers' scripts.

  2. Why the simple system fails

    Unit and integration tests check one service, one request at a time, with inputs you thought of. Bugs between services, under concurrency and from hostile input slip through.

  3. Engineering concept

    Contracts verify the seams between services; load tests verify behaviour under realistic and extreme traffic; security tests hunt for inputs nobody planned for.

  4. Technology

    Pact-style consumer-driven contracts, schema compatibility checks, k6 load scripts with p95/p99 thresholds, SAST, DAST, dependency scanning and fuzzing.

  5. Trade-offs

    Contracts need cross-team discipline; load tests need prod-like environments and time; security scanners produce false positives. Each costs more than a unit test, so aim them precisely.

01

The problem

2 min
Monday, 9:00. Planning the next three weeks.
  1. Omar · On-call SRE
    Booking renames guestName → guest_name in Wednesday's release. Its tests all pass.
  2. Maya · Product manager
    Email and invoicing read that field. Their tests pass too. And Black Friday brings 4× the usual search traffic in three weeks.
  3. Lina · Senior engineer, your mentor
    You lead this one. Start with the rename.
02

The big idea

4 min
Two green suites, one broken email
The big idea

Test the seams unit tests can't see: contracts check what each service expects of the other; load tests check what happens when thousands of guests arrive at once.

Building a bridge, you check that supplier A's bolts fit supplier B's holes before either ships. Then you drive the expected trucks across, and keep adding trucks until something bends.

03

How it works

5 min
Two seams, two kinds of test
  1. 1
    Consumers write contracts

    A : email records the requests it makes and only the fields it reads (guestName, email, checkIn), and publishes it to a broker.

  2. 2
    The provider verifies

    Booking's CI replays every consumer's contract before deploying. The rename fails in minutes, naming email and invoicing.

  3. 3
    Expand, migrate, contract

    Add guest_name and keep sending guestName; consumers switch; remove guestName when no contract reads it. A rename breaks backward compatibility (new code reads old data) and forward (old code tolerates new data).

  4. 4
    Load and stress

    A holds expected peak (600 req/s for 15 min). A stress test keeps adding traffic to find the breaking point and how you fail.

  5. 5
    Spike and soak

    Spike: jump 10× in seconds (does autoscaling keep up?). Soak: normal load for hours (memory leaks, slow drift).

  6. 6
    Thresholds decide

    A threshold is a pass/fail rule: p99 under 1 s and errors under 1%. Never the average: a 142 ms mean can hide a 2.9 s .

04

Watch a decision being made

4 min
Worked example

You prepare search for Black Friday, Lina checks

Search must meet its SLO (under 1 s, no 5xx) at the forecast peak of 600 searches a second, 4× today. Find out whether it does, what breaks first, and how much margin is left.

  1. 1Write the question as thresholds, prod-like

    At 600 req/s for 15 minutes: p99 < 1 s, errors < 1%. Same instance sizes, pool and data volume as production, spun up only for the test.

    Why: Lina: "Without thresholds, a load test is a chart people argue about. A tiny box finds the test environment's limits."

  2. 2Make traffic look like guests

    Cities and dates drawn from last year's search logs, 1–4 s of think time, a realistic mix of search, hotel page and checkout.

    Why: Repeating one URL measures one cache entry.

  3. 3Read percentiles and saturation together

    Result: avg 142 ms, p99 2.9 s, errors 0.4%. App CPU 40%. DB pool: 20 connections busy, ~180 requests waiting.

    Why: High latency with idle CPU means waiting. The resource with a queue in front of it is the bottleneck, so more app servers would only add waiters.

  4. 4Fix, rerun, measure the margin

    Cache popular searches 60 s, pool 20 → 40 behind PgBouncer, index on availability(city, date). Rerun: p99 380 ms. Stress: breaking point ~1,500 req/s. Soak 8 h: memory flat.

    Why: Passing at the forecast isn't enough; you need to know how much is left.

The whole thing in one line: SLO as thresholds, guest-like traffic on prod-like hardware, p99 read next to saturation, then fix what queued and measure the margin.
05

In real life

3 min
Consumer-driven vs provider-owned

Pact: consumers write the contract

  • Open-source tool that publicly traces its origins to the property site realestate.com.au
  • Each consumer records what it needs; the provider verifies all of them

Fits HTTP APIs between teams that deploy independently.

Confluent Schema Registry: rules on the schema

  • Widely used with Kafka: producers register event schemas centrally
  • Rejects versions that break the compatibility mode; doesn't know who reads which field

Fits events with many unknown or future readers.

06

What goes wrong

3 min
No think time, one URL repeated

It warms one cache entry and makes everything look fast.

Instead

A realistic mix, think time, parameters from real logs.

Reporting the average latency

A 142 ms mean hid a 2.9 s p99.

Instead

Thresholds on p95 and p99 per endpoint, plus the error rate.

Updating the consumer's stub in the same change as the rename

Both suites stay green and the seam stays untested.

Instead

The provider verifies the consumers' published contracts.

07

Trade-offs

3 min
OptionFits whenYou pay
Consumer-driven contractsTeams deploy independentlyA workflow for every team; checks shape, not full behaviour
Shared staging environmentA few journeys as a final smoke checkSlow, contended, often red; blocks independent deploys
Load tests in a prod-like environmentA known peak, latency SLOs~€4,000/month while running; noisy results
Tuesday, 10:00. How to catch cross-service breaks from now on.
  1. Karim · CTO
    Keep the shared staging environment and test everything together before releases. No new tools.
  2. Omar · On-call SRE
    Staging found last month's rename three days late. And it's red half the time for someone else's change.
  3. Karim · CTO
    Contracts mean every team learns a broker and a new workflow.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • How a rename ships: expand, migrate, contract over several releases, not one Wednesday deploy.
  • The load test target and pass rule: hold 600 req/s with under 1 s and errors under 1%, then push until it breaks.
  • What "realistic" means: varied cities and dates, think time, production-sized data, a prod-like environment spun up only for test runs.
Tell Claude Code
Write a k6 [[load-test]] for search, plan first. Mix: 80% search, 15% hotel pages, 5% booking starts, 1–4 s think time. Pick cities and dates from the CSV of last month's real searches in perf/data. Ramp to 600 req/s over 5 minutes, hold 15, then add 150 req/s every 3 minutes until errors pass 1%. Thresholds: p99 under 1 s, errors under 1% during the hold. Target only the perf URL from config; refuse to run against production.
Check before you accept
  • Cities and dates vary and there's think time; one fixed city only measures the cache.
  • Thresholds name p99 and error rate; if the report leads with the average, ask for percentiles.
  • The script refuses a production URL: ask Claude to show you that line.
Red flags
  • It calls the test passed on a 142 ms average while p99 is 2.9 s.
  • It "fixes" slow search by raising the k6 timeout or lowering the target.
  • It runs the test against production or shared staging without asking.
09

Check yourself

2 min
Q1/3Debugging
Booking service renames guestName to guest_name. Booking's tests pass. Email service's tests pass (they use a stub of the booking API). Production emails say 'Dear undefined'. Which test would have failed first, in which pipeline?
How confident are you?

Next episode: the September cloud bill arrives at €310k, and it grew much faster than bookings did.

Finished?

Mark it complete to update your skill map.

Next: The cloud cost model: what you actually pay for
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

What breaks at four times the traffic
Worked example

You prepare search for Black Friday, Lina checks

Search must meet its SLO (good = under 1 s, no 5xx) at the forecast Black Friday peak of 600 searches a second, 4× today. Find out whether it does, what breaks first, and how much margin there is.

  1. 1Write the question as thresholds

    At 600 req/s for 15 minutes: p99 < 1 s, p95 < 400 ms, errors < 1%.

    Why: Lina: "Good: the SLO becomes a pass/fail gate. Without thresholds, a load test is a chart people argue about."

  2. 2Use a prod-like environment

    Same instance sizes, pool settings and data volume as production (a copy of availability data), spun up only for the test.

    Why: A smaller box or a tiny database finds the test environment's limits, not production's.

  3. 3Make traffic look like guests

    Cities, dates and nights drawn from last year's search logs; think time between requests; search, hotel page and checkout in a realistic mix.

    ts
    export default function () {
  const s = pickFromProductionMix();            // city, check-in, nights
  http.get(`${__ENV.BASE_URL}/search?city=${s.city}&checkin=${s.checkIn}&nights=${s.nights}`,
    { tags: { name: "search" } });
  sleep(1 + Math.random() * 3);                 // guests pause between clicks
}

    Why: Repeating one URL measures one cache entry. Varied searches exercise the database the way Black Friday will.

  4. 4Read percentiles and saturation together

    Result: avg 142 ms, p99 2.9 s, errors 0.4%. App CPU 40%. DB pool: 20 connections busy, ~180 requests waiting.

    Why: High latency with idle CPU means waiting. The resource with a queue in front of it is the bottleneck.

  5. 5Fix the bottleneck, then rerun

    Cache popular searches for 60 s, pool 20 → 40 behind a connection pooler, add an index on availability(city, date). Rerun: p99 380 ms.

    Why: Each fix targets what saturated. Rerunning proves it, and shows where the next bottleneck is.

  6. 6Find the margin and the slow failures

    Stress until errors appear: breaking point ~1,500 req/s (2.5× forecast). Soak at 300 req/s for 8 h: memory flat. Spike 150 → 1,000 in 30 s: autoscaling lags 90 s, so pre-scale on the morning of Black Friday.

    Why: Passing at the forecast isn't enough. You need to know how much is left, and what happens in the first minute of a surge.

The whole thing in one line: You turned the SLO into thresholds, ran prod-like, guest-like traffic, read p99 next to saturation to find the database pool, fixed it and reran, then measured the margin with stress, soak and spike tests.
11

Interactive simulation

12

Visual diagram

Click a component

Top: a consumer-driven contract flows from consumer to provider CI. Bottom: a load test hits a prod-like environment while monitoring records percentiles.

13

Analogy

  • Building a bridgeSupplier A's bolts fit supplier B's holesTesting a systemContract test between services
  • Building a bridgeDrive the expected number of trucks acrossTesting a systemLoad test at expected peak
  • Building a bridgeKeep adding trucks until something bendsTesting a systemStress test: find the limit
  • Building a bridgeLeave traffic running for weeksTesting a systemSoak test: find slow leaks
  • Building a bridgeAn inspector hunting for weak spotsTesting a systemSecurity testing

Where the analogy breaks: a bridge changes rarely. Your services change every day, so these tests must run automatically, on every change.

14

Check your understanding

Q1/4Reasoning
Classify each change to a booking event schema read by several consumers:
Add an optional specialRequests field
Rename checkIn to check_in
Change totalCents from integer to string
Make the optional promoCode required
How confident are you?
15

Practice ladder

Step 1 · Trace

Which provider changes break which consumer?

Scenario Warm-up

Contracts on the broker for GET /bookings/:id:

  • email reads: guestName (string), email (email), checkIn (date)
  • invoicing reads: guestName (string), totalCents (integer), currency (string)

Trace each proposed booking change: which contract fails, if any?

  1. Add a new field loyaltyTier.
  2. Remove roomView (nobody lists it).
  3. Change totalCents from integer to a decimal total in euros.
  4. Rename guestName to guest_name.
  5. Send checkIn as "10/11/2026" instead of "2026-11-10".
Step 2 · Complete

Finish the rename plan

Design Practice

Your plan to rename guestName → guest_name has two gaps. Fill in 2 and 4.

  1. Expand (booking, release 1): add guest_name; keep sending guestName with the same value. All contracts still pass.
  2. ___
  3. Check: the broker shows no consumer version in production still reads guestName.
  4. ___
  5. For the booking.created event: ___ (one line)
Step 3 · Solve

Load-test checkout before Black Friday

Design Practice

Search is ready. Now checkout: normal peak ~12 checkouts/s, Black Friday forecast 50/s, with a flash sale at 10:00 that Maya expects to bring 200/s for about 2 minutes. Checkout calls the payment provider (whose sandbox allows only 20 requests/s) and writes to the bookings database.

Design the tests: which types, what traffic, what thresholds, and how you handle the payment provider.

Step 4 · Decide

Load test in production at 3 a.m.?

Decision Stretch

Karim: "Spinning up a prod-like perf environment costs ~€4,000 a month while it runs. Let's load-test production itself at 3 a.m., when traffic is low. It's the real environment, for free."

Facts: at 3 a.m. there are still ~20 searches/s from Asia; tests would add 600/s; search shares its database with booking; the payment provider can't be load-tested.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain to a teammate why 'all our services have 90% coverage' doesn't mean a renamed field, a Black Friday peak or a stored XSS will be caught, and what you would add for each.

0 words