Episode 70 · Slow alone, slow together

Performance vs scalability

Monday, 9:30: search takes 1.8 s for a single guest at 3 a.m., and checkout takes 0.2 s at night but 3.1 s every Friday at 19:00. Karim wants to add ten servers to fix both.

Fast for one person is not the same as fast for a million. Learn to tell the two apart, because the fixes are completely different.

30 min intermediateLatencyThroughputPerformanceScalabilityp50 / p95 / p99 latencyFinding the bottleneck

Problem
The hotel search page is quick in testing but crawls every Friday evening when 20× more guests arrive.
Why the simple system fails
One server and one database have a fixed capacity. Past it, requests queue, latency explodes, and making code faster only delays the wall.
Engineering concept
Performance is how fast one request is; scalability is how well the system keeps that speed as load grows. Find the bottleneck, then add capacity where it is.
Technology
Percentile monitoring, load testing, stateless app servers behind a load balancer, caches, read replicas, sharding.
Trade-offs
Every scaling step adds cost, moving parts and consistency problems. Scale the bottleneck you have, not the one you imagine.

The problem

2 min

Monday, 9:30. StayHub is heading for a million guests a day.

Maya · Product manager
Two complaints. Search takes 1.8 s, even at 3 a.m. Checkout is 0.2 s at night but 3.1 s every Friday at 19:00.
Karim · CTO
Add 10 servers. About €3,000 a month. That should fix both.
Lina · Senior engineer, your mentor
Maybe. Before we spend it, make a guess.

The big idea

4 min

Two slow pages, two different problems

The big idea

Slow alone is a performance problem; slow only when busy is a scalability problem. Ask which one before you choose a fix.

Search: one joined query instead of 51 takes it from 1.8 s to about 250 ms, for free. Checkout needs something else entirely. Neither needs ten servers.

How it works

5 min

Where a request's time goes

1
Working
The server runs code: parsing, ranking, rendering. Shows up as CPU. Latency is how long one request takes, click to answer.
2
Waiting on others
Queries, other services, the payment provider. CPU sits idle while it waits.
3
Waiting in line
Under load, requests queue for something shared and limited: database connections, locks, worker threads. This caps throughput, the requests completed per second.
4
Alone: work + waiting on others
Search at 3 a.m.: 50 price queries one after another, 1.5 s of the 1.8 s. A problem: fix the slow step.
5
Busy: + waiting in line
Checkout on Friday: requests queue for 20 database connections. A problem: find the shared .
6
Little's law
Max throughput = slots ÷ time each request holds one. 20 connections held 150 ms each: 20 ÷ 0.15 ≈ 133 requests/s. Friday brings 180.

Watch a decision being made

4 min

Worked example

Lina finds checkout's wall

Checkout's p95 is 3.1 s on Friday at 19:00 (~180 requests/s) with app CPU at 22%. Find the limit and remove it, without adding servers.

1Find what they wait for
A trace of a slow checkout: 2.8 s spent waiting for connection from pool before the first query. The pools hold 20 connections (4 pods × 5).
Why: Pool and queue metrics show waiting that CPU graphs can't. The slow step isn't a query; it's the line in front of it.
2Measure how long each request holds a slot
About 150 ms, because a ~120 ms call to the payment provider runs inside the database transaction.
ts
```
await db.transaction(async (tx) => {
  await tx.lockRoom(roomId);
  await payments.charge(card);   // holds the connection ~120 ms
  await tx.saveBooking(booking);
});
```
Why: Capacity depends on how long each request occupies the scarce thing, not only on how many there are.
3Compute the ceiling
20 ÷ 0.15 s ≈ 133 requests/s. Friday brings 180, so the line grows every second.
Why: Little's law turns "slow on Fridays" into a number you can compare with demand.
4Shorten the hold, not the line
Charge the card (with an idempotency key) before opening the transaction, then lock and save: about 30 ms per connection. 20 ÷ 0.03 ≈ 660/s. Next Friday: p95 240 ms.
Why: Doubling the pool to 40 gives ~266/s and doubles database load. Shorter holds beat more slots.

In real life

3 min

Make each server do more, or add more servers?

WhatsApp: a lot from each server

Publicly described as serving huge numbers of users with a small team
Engineers talked publicly about tuning each server to hold millions of connections
Fewer machines, deep performance expertise

Worth it when one workload dominates and a small team knows it deeply.

Netflix: many instances, scaled out

Has publicly described running services on large fleets of cloud instances
Adds and removes instances with demand
Needs stateless services, automation and tooling

Worth it for many services and big swings in demand.

What goes wrong

3 min

Scaling the app tier when the database is the bottleneck

More servers send more connections and queries to the same database.

Instead

Relieve the saturated resource: shorter transactions, pooling, caching, query fixes.

Adding servers to a slow single request

They run the same slow code in parallel.

Instead

Trace one request first and fix the slow step.

Holding scarce resources during slow calls

Your capacity now depends on someone else's latency.

Instead

Make outside calls before or after the transaction.

Trade-offs

3 min

Lever	Helps when	Catch
Fix the slow step	A trace shows one slow step or a long hold	Engineering time; risk in critical code
Bigger machine	Need headroom fast on a stateful part	~€2,000/month here; hard ceiling; cause stays
More machines	Each server is the limit, tier is stateless	Useless against shared bottlenecks

Monday, 15:00. Friday is four days away.

Karim · CTO
Your fix changes checkout, our most sensitive code, four days before Friday. A bigger database is one click.
Lina · Senior engineer, your mentor
About €2,000 more a month, and each request still holds a connection during the payment call.
Karim · CTO
Money I have. A broken checkout on Friday I don't.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which problem you're solving: slow for one guest at 3 a.m. (performance) or slow only at Friday 19:00 (scalability).
The target, in numbers: search under ~250 ms alone; checkout p95 under a second at ~180 requests/s.
What a change may cost the shared bottleneck: all pods' connections must fit in what's left of Postgres's 200.

Tell Claude Code

Checkout p95 hits 3.1 s every Friday at 19:00 (~180 requests/s) while app CPU sits at 22%. Before changing anything, investigate and report: how long each checkout request holds a database connection, what happens inside the transaction, pool sizes (5 per pod, 20 total), and how close we are to Postgres's 200. If outside calls run inside the transaction, propose moving them out, and how we stay consistent if payment fails. Don't add pods or change pool sizes. Then load-test staging at 200 requests/s: p95 and hold time, before and after.

Check before you accept

The plan names the bottleneck with evidence (hold time, pool wait), not just "it's slow".
Little's law, yourself: 20 ÷ 0.15 s ≈ 133/s. Does the fix move that limit?
The load test compares p95 at the Friday rate, not the average at low load.

Red flags

It adds pods while CPU is low and requests wait on something shared.
It raises the pool size or max_connections without asking what holds the connections.
It reports average latency or a single run as proof.

Check yourself

2 min

Q1/3Debugging

Traffic doubled this month. API p95 latency went from 100 ms to 2 s, but app-server CPU is only at 20%. What do you check first?

Setup: 4 app servers behind a load balancer, one PostgreSQL primary, each app server has a DB connection pool of 10.

How confident are you?

Next episode: the dashboard says search averages 120 ms, yet one in three search pages feels slow. Omar suspects the average is lying.

Finished?

Mark it complete to update your skill map.

Next: Latency percentiles and the tail

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Why checkout hits a wall at 133 per second

Lina diagnoses checkout, thinking out loud. The tool is Little's law: concurrency = throughput × latency. It tells you the most a shared resource can handle.

Worked example

Lina finds checkout's wall

Checkout's p95 is 0.2 s at night and 3.1 s on Friday at 19:00 (~180 checkout API requests/s: about 5 booking forms are opened per second, roughly 3 per completed booking, and each open form makes ~35 calls: availability re-checked every few seconds, a price quote per date change, then payment and confirmation), with app CPU at 22%. Find the limit and remove it, without adding servers.

1Rule out working
App CPU 22%, memory fine. The pods aren't busy computing.
Why: Low CPU with high latency means the time goes to waiting. Speeding up code won't help.
2Find what they wait for
A trace of a slow checkout shows 2.8 s spent before the first query even starts: waiting for connection from pool. The pools hold 20 connections in total (4 pods × 5).
Why: Pool and queue metrics show waiting that CPU graphs can't. The slow step isn't a query; it's the line in front of it.
3Measure how long each request holds a slot
Each checkout request holds a connection for about 150 ms, because its call to an outside service (~120 ms: the payment provider here, the hotels' rate feed for price and availability) runs inside the database transaction.
ts
```
await db.transaction(async (tx) => {
  await tx.lockRoom(roomId);
  await payments.charge(card);   // holds the connection ~120 ms
  await tx.saveBooking(booking);
});
```
Why: Capacity depends on how long each request occupies the scarce thing, not just on how many things you have.
4Compute the ceiling
20 connections ÷ 0.15 s = about 133 requests/s. Friday brings 180. Everything above 133 waits in line, and the line grows every second.
Why: Little's law turns "it's slow on Fridays" into a number you can compare with demand.
5Shorten the hold, not the line
Make the outside call before opening the transaction (charge the card with an idempotency key, fetch the rate), then lock the room and save in a short transaction: about 30 ms per connection.
Why: 20 ÷ 0.03 s ≈ 660 requests/s: almost 4× Friday's peak, with no new servers and no bigger database.
6Measure again
Next Friday: p95 at 19:00 is 240 ms, pool wait near zero, database CPU unchanged.
Why: Re-measuring confirms the fix and shows where the next limit is before it finds you.

Lever	Improves	Typical gain	Cost / catch
Fix a slow query (index, no N+1)	Performance (latency)	500 ms → 5 ms on one query	Engineering time; slower writes for indexes
Vertical scaling (bigger machine)	Capacity	~2× per size step	Hard ceiling, still one machine
Horizontal scaling (more servers)	Throughput	Near-linear for stateless tiers	Doesn't help shared bottlenecks
Caching	Both	10–100× on hot reads	Stale data, invalidation bugs
Read replicas	Read throughput	+1× reads per replica	Replication lag
Connection pooling	DB throughput	Thousands of clients on ~100 DB connections	One more component to run

Each lever targets a different problem. Pick the one that matches your bottleneck.

Interactive simulation

Visual diagram

Guests

Load balancer

App server 1

App server 2

App server 3

Redis cache

PostgreSQL primary

Read replica

Click a component

A hotel search system after its first scaling steps. Walk through where the capacity limits are.

Analogy

A hotel check-in deskA software system

A hotel check-in deskOne guest takes 3 minutesA software systemLatency
A hotel check-in deskThe desk handles 20 guests an hourA software systemThroughput
A hotel check-in deskPre-filled forms, faster card machineA software systemPerformance work
A hotel check-in deskOpening more desks for the 6 pm rushA software systemScalability work
A hotel check-in deskOne key-card printer for all desksA software systemThe bottleneck

Where the analogy breaks: hotel guests queue politely. Overloaded software often gets slower for everyone at once, and can even start failing, as waiting work piles up.

Check your understanding

Q1/4Reasoning

Match each observation to the most likely diagnosis:

Search takes 3 s even at 3 a.m. with one user

Search takes 80 ms alone but 4 s during the Friday peak

p50 is 90 ms but p99 is 6 s at normal load

Going from 4 to 8 app servers didn't raise throughput

How confident are you?

Practice ladder

Step 1 · Trace

Performance or scalability?

Scenario Warm-up

For each observation, say whether it's mainly a performance or a scalability problem, and what you'd look at first.

The owner dashboard takes 4 s to load, at any hour, even when only one owner is logged in.
The reviews API answers in 80 ms at night and 1.5 s at lunchtime; its CPU is 25% at lunchtime.
The photo-resize worker's queue grows every evening; each worker is at 98% CPU.
Hotel search is 300 ms for everyone, but 5% of searches time out on Sunday evenings.

Step 2 · Complete

Finish Lina's capacity note

Estimation Practice

Lina's note on the owner-messages API has gaps.

It runs 4 pods; each pod has 25 worker slots (concurrent requests).
Each request holds a slot for 80 ms on average.
Peak demand on Monday mornings: 1,500 requests/s.

Total slots: ___
Maximum throughput: ___ requests/s
Is Monday's peak above or below the ceiling? What happens to latency?
Option A: double the pods. Option B: cut the hold time to 40 ms by removing a duplicate query. Compute the new ceiling for each.

Step 3 · Solve

The slow owner dashboard

Debugging Practice

Owners say their dashboard takes 4 s, at any hour. One trace (one owner, 3 a.m.):

GET /owner/dashboard                 4,020 ms
  auth check                            12 ms
  SELECT hotels WHERE owner_id=…        18 ms   (returns 25 hotels)
  for each hotel:
    SELECT bookings this month         ~95 ms   x 25
    SELECT reviews last 30 days        ~55 ms   x 25
  render                                40 ms

Karim offers to double the dashboard's pods. Explain why that won't help, find the cause, and propose fixes with expected gains.

Step 4 · Decide

Scale up search's database?

Decision Stretch

After the N+1 fix, search is fast alone (250 ms). But on Sunday evenings search p95 reaches 1.2 s. Omar's data:

Search pods: CPU 35%.
Postgres primary: CPU 88% at Sunday peak, mostly search queries (reads).
Writes (bookings) are about 5% of database load.
A database one size up: ~€2,000 more a month, ~10 minutes of downtime to resize.
A read replica for search: ~€1,800 a month (same size as the primary); search would read data up to ~1 s old.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

A product manager says: "The site is slow, let's add more servers." Explain how you would respond and why.

0 words