Episode 8 · Before you buy a bigger server

CPU, memory, disk and network: finding the bottleneck

Thursday, 10:00: hotel search takes 2.4 seconds at peak, and Karim wants to double the app servers for €600 a month before summer. Lina says you lead this one: find what's actually slow first.

Every request uses four resources, and they differ in speed enormously. Learn which one is really slowing you down before you try to fix anything.

30 min foundationThe latency ladder: cache, RAM, SSD, networkCPU-bound vs I/O-bound workServer resources: CPU, memory, disk, networkFinding the bottleneckLatencyThroughput

Problem
The booking API is slow and the team wants to buy a bigger server.
Why the simple system fails
Without knowing which resource is saturated, a bigger machine often fixes nothing: doubling CPU doesn't help a request that spends 95% of its time waiting on the network.
Engineering concept
Each resource has a characteristic speed (the latency ladder) and a capacity limit. Measure utilisation, saturation and errors per resource, then fix the one that limits.
Technology
CPU cores, RAM and CPU caches, SSDs and IOPS, network RTT and bandwidth; tools like top, vmstat, iostat and request tracing.
Trade-offs
Moving data up the ladder (caching in RAM, co-locating services) buys speed with cost, memory pressure and staleness.

The problem

2 min

Thursday, 10:00. Six weeks before summer.

Maya · Product manager
Hotel search: 2.4 s at peak. Competitors answer in half a second.
Karim · CTO
Double the app servers. €600 a month. I can approve it today.
Lina · Senior engineer, your mentor
You lead this one. Tell Karim what to buy, if anything.

The big idea

4 min

Where do 2.4 seconds go?

The big idea

Find the resource that's full before buying more of anything.

CPU, memory, disk and network differ in speed by factors of a thousand. Whichever one is full sets the speed for everything else.

How it works

5 min

Every request uses some (compute), (hold), disk (keep) and network (talk). The latency ladder: CPU cache ~1 ns, RAM ~100 ns, SSD ~tens of µs, a network round trip 0.5–100 ms. For each resource, ask how busy it is and whether work is queued for it: a growing queue (saturation) means it's full.

Your checks, in order

1
Trace one request
Where does the time go? 31 ms CPU in the app, 2.1 s waiting on the database.
2
App CPU
35% average, no core pegged, nothing waiting for a core. Not CPU.
3
App memory
Stable at 60%, no restarts. Not memory.
4
Database machine
CPU 40%. Memory full: its 16 GB of RAM holds only part of the 22 GB availability data.
5
Database disk
Flat at the 3,000 cap (reads and writes per second), 48 ms per read, a queue of 31. Saturated.

The chain: data outgrew RAM, so reads fall to the SSD; the search query scans thousands of pages; the disk hits its cap and reads queue for 48 ms. Dozens per search make 2.1 s. The is disk reads on the database.

Watch a decision being made

4 min

Worked example

You find the 2.4 seconds (Lina checks)

Search p95 is 2.4 s at peak; the target is 500 ms. Karim offers €600 a month for twice the app servers. Find the bottleneck and recommend what to buy.

1Rule out the app servers
The trace points at the database. vmstat on an app server: 2 threads waiting for 8 cores, 65% idle.
Why: Lina's check: "Right. A resource with no queue isn't your bottleneck, however busy it looks."
2Check the database disk
Memory full (22 GB of data, 16 GB of RAM). Then the disk:
bash
```
$ iostat -x 1
Device   r/s    w/s  aqu-sz  r_await  %util
nvme1n1  2980   21   31.4    48.2     100.0
```
Why: Lina's check: "Flat at the 3,000 IOPS cap with a queue of 31. That's saturation."
3Name both causes
The query reads thousands of pages, and since the data outgrew RAM, most come from disk.
Why: Lina's check: "Two causes, and they multiply: too many pages, too few in memory."
4Fix each cause, cheapest first, and measure
An index on availability(hotel_id, night): p95 ~400 ms. Database RAM 16 → 32 GB (+€150/month): p95 ~180 ms.
Why: Lina's check: "€150, not €600, aimed at the machine that's actually full."

In real life

3 min

Two well-known databases, two places to keep data

Redis: everything in RAM

Its documentation describes keeping the whole dataset in memory
Reads and writes take microseconds
Data can't outgrow the RAM you pay for

Fits small, hot data where speed beats size.

PostgreSQL: on disk, RAM as a cache

Its documentation describes tables on disk, recently used pages kept in memory
Data can be far bigger than RAM, and survives restarts
When the hot part outgrows RAM, reads fall to disk

Fits large, durable data, if the hot part fits in RAM.

What goes wrong

3 min

Scaling up before measuring

If disk, network or another service is the bottleneck, more app CPU changes nothing.

Instead

Trace a slow request; check how busy and how queued each resource is.

Loading a whole table into memory

Memory grows with your data until the process is killed.

Instead

Stream or paginate in batches.

Confusing bandwidth with latency

A faster link doesn't shorten a round trip; 50 small calls in a row are limited by latency.

Instead

Cut round trips (batch, run in parallel); add bandwidth only for big transfers.

Trade-offs

3 min

Where data lives	Fits	You pay	Main risk
Process memory	Config, feature flags	RAM × every process	Copies disagree
Shared cache (Redis)	Read often, slight staleness fine	A network hop, another system	Stale data
Database (RAM as cache)	Must be fresh: availability, payments	Enough RAM for the hot part	Slow when hot data falls to disk

Monday, 10:00. The index is in; p95 is ~400 ms.

Karim · CTO
Cache search results in Redis for 60 seconds. No new hardware.
Omar · On-call SRE
Availability changes every minute. A 60-second-old "2 rooms left" sells rooms that are gone. I'd pay €150 for database RAM.
Karim · CTO
€150 every month, and it only buys time until the data grows again.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What you spend money on: +€600 for app servers or +€150 for database RAM, approved after seeing where the time goes.
The target that defines "fixed": search p95 under 500 ms, measured the same way before and after.
Which big jobs must stream data instead of loading it all (the CSV export of 1.2M bookings).

Tell Claude Code

Search p95 is 2.4 s; target 500 ms. A trace shows 31 ms of app CPU and 2.1 s waiting on PostgreSQL. The database disk is flat at its 3,000 IOPS cap with 48 ms read waits, and ~22 GB of availability data no longer fits in 16 GB of RAM. Don't add app servers. Show the search query's EXPLAIN ANALYZE, propose an index (we think hotel_id plus night), and estimate moving to 32 GB of RAM. Create the index only on staging; report p95 and IOPS before and after.

Check before you accept

The plan names the saturated resource with a number (3,000 IOPS, 48 ms waits), not "the database is slow".
The query plan after the change uses the new index instead of reading most of the table.
On staging, p95 is under 500 ms and IOPS well below the cap.

Red flags

More app servers or a faster language while app CPU sits at 35%.
It raises the Node memory limit to make exit code 137 go away.
An improvement quoted without a measurement ("should be 3× faster").

Check yourself

2 min

Q1/3Debugging

A reporting API is slow. API host: CPU 15%, memory stable. Database host: CPU 30%, iostat shows %util 100, await 45 ms, IOPS flat at 3,000. What's the bottleneck?

How confident are you?

Next episode: a €156 stay shows a checkout total of €12,036 on the new app, with no error anywhere. Time to look at types and functions.

Finished?

Mark it complete to update your skill map.

Next: Types, functions and control flow

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Reading the four resources

A simple method for each resource is USE: check Utilisation (how busy), Saturation (how much work is queued) and Errors. Saturation is the key signal: a resource at 70% can be fine, or have a growing queue.

Worked example

You find the 2.4 seconds (Lina checks)

Hotel search p95 is 2.4 s at peak; the target is 500 ms. Karim offers €600 a month for twice the app servers. Find the bottleneck and recommend what to buy, if anything.

1Trace one slow request
Pick a slow search from the traces: 31 ms CPU in the app, 2.1 s waiting on the database, the rest network and rendering.
Why: Lina's check: "Good. One trace tells you which machine to look at. Without it, you'd be tuning blind."
2Rule out the app servers
vmstat on an app server: 2 threads waiting for 8 cores, 65% idle. Memory stable, no restarts.
Why: Lina's check: "Right. A resource with no queue isn't your bottleneck, however busy it looks."
3Check the database machine, resource by resource
CPU 40%. Memory: full, with the 22 GB availability data larger than the 16 GB of RAM. Disk:
bash
```
$ iostat -x 1
Device   r/s    w/s  aqu-sz  r_await  %util
nvme1n1  2980   21   31.4    48.2     100.0
```
Why: Lina's check: "Yes: flat at the 3,000 IOPS cap with a queue of 31. That's saturation, and it explains the 48 ms per read."
4Name the cause
The search query scans thousands of availability pages. Since the data outgrew RAM, most of those pages come from disk, and the disk can't keep up.
Why: Lina's check: "Two causes, and they multiply: too many pages per search, and too few of them in memory."
5Pick the fixes, cheapest first
1. An index on availability(hotel_id, night): each search reads a few dozen pages. 2. Database RAM 16 → 32 GB (+€150/month) so the hot data fits. Expected p95: ~400 ms, then ~180 ms.
Why: Lina's check: "Each fix attacks one cause. And it's €150, not €600, aimed at the machine that's actually full."
6Measure again
After the index: disk at ~20% of its cap, queue near 0. After the RAM: p95 180 ms at peak.
Why: Lina's check: "Good. Now the next bottleneck is somewhere else, and you'll find it the same way."

Inside your code, the most common hidden cost is waiting several times in a row. The same work, done in parallel, costs the slowest call instead of the sum:

// One after another: the waits add up (≈ 1 + 1 + 90 ms)
const hotel = await db.hotel(id);
const rooms = await db.rooms(id);
const rate = await partner.rate(id); // another region

// At the same time: the slowest one decides (≈ 90 ms)
const [hotel2, rooms2, rate2] = await Promise.all([db.hotel(id), db.rooms(id), partner.rate(id)]);

Latency is mostly how many times you wait, not how fast you compute.

Resource	Full looks like	Typical fixes
CPU	Threads waiting for a core; a core at 100%; slower under load even without I/O	Profile hot code, cache results, more cores, move heavy work to background jobs
Memory	Swapping, processes killed (exit 137), cache hit rate falling	Fix leaks, stream instead of loading everything, bigger machine
Disk	High I/O wait, a queue, tens of ms per read, IOPS flat at the cap	Indexes, more RAM for caching, a higher-IOPS disk, batching writes
Network	Many calls in a row, large payloads, far-away hops, bandwidth flat at the link's limit	Batch and parallelise calls, compress, paginate, move things closer

Interactive simulation

Visual diagram

Guest app (Lisbon)

Booking API (CPU)

In-process cache (RAM)

Redis (network + RAM)

PostgreSQL

SSD volume (IOPS cap)

Payment provider

Click a component

Where one 180 ms booking request actually spends its time.

Analogy

In a kitchenIn a server

In a kitchenThe chef's handsIn a serverThe CPU
In a kitchenThe cutting boardIn a serverThe CPU cache
In a kitchenThe counter, one step awayIn a serverMemory (RAM)
In a kitchenThe walk-in fridge at the backIn a serverThe disk
In a kitchenOrdering from a market across townIn a serverA call over the network

If dinner is late, a faster chef rarely helps. Usually each ingredient was fetched from the market one at a time. Fetch in bulk, keep favourites on the counter, or move closer to the market. Where the analogy breaks: computers can fetch many things at once, in parallel.

Check your understanding

Q1/3Recall

Order from fastest to slowest:

1.RAM access
2.Round trip Europe ↔ US
3.Round trip within one data centre
4.CPU L1 cache hit
5.Random 4 KB read from an SSD

How confident are you?

Practice ladder

Step 1 · Trace

Trace where a booking page waits

Estimation Warm-up

The booking confirmation page does this, one step after another:

3 ms of CPU to check the input
1 query to PostgreSQL in the same data centre: 0.5 ms round trip + 2 ms in the database
40 separate Redis GETs, one per extra (breakfast, parking…): 0.5 ms round trip each
A call to the payment provider in another region: 90 ms round trip + 200 ms of their work
4 ms of CPU to build the response

Add it up. Which resource (CPU, memory, disk, network) does most of the time go to, and which single step would you fix first?

Step 2 · Complete

Finish your note on the slow owner dashboard

Debugging Practice

The owner dashboard takes 3 s to load. Your investigation so far:

Trace one slow request: 2.8 s of CPU in the app process, 40 ms waiting on the database.
Rule out the database: ___
Check the app server, resource by resource: vmstat shows r = 12 on 8 cores, 2% idle. Memory stable. No disk waits.
Name the cause: ___
Pick a fix: ___

Fill in steps 2, 4 and 5. The dashboard recomputes each hotel's yearly statistics from scratch on every page load.

Step 3 · Solve

The nightly job that slows the morning

Debugging Practice

Every night at 02:00, a job copies the hotels' latest availability into StayHub's availability table. Since last month, the job runs until 08:30 and guest searches between 07:00 and 08:30 are slow.

What Omar sees during the job: - Database CPU: 25% - Database disk: writes at ~2,900 per second, near the 3,000 IOPS cap; read latency 35 ms - The job updates one row per statement (one room, one night), committing after each one: about 20 million updates per night - Search p95 at 07:30: 1.6 s

Find the bottleneck, explain why searches suffer, and propose fixes.

Step 4 · Decide

Summer: what do we buy?

Decision Stretch

Summer traffic is expected at 3× today's peak. Your measurements at today's peak, after the index and the RAM upgrade:

App servers: CPU 22%, memory 55%, no queue
Database: CPU 35%, RAM 32 GB with the hot data at 24 GB, disk at 15% of its IOPS cap
Payment provider calls: 290 ms each, one per booking
Budget: up to €600 a month

Options: A double the app servers (€600). B a bigger database machine, 16 cores and 64 GB (€450). C nothing now; a load test at 3× first, then buy what it shows (about €50 of test machines).

Write a short decision: "We chose X over Y because…, accepting the risk of…"

Explain it yourself

Your manager says: “The API is slow, let's double the server size.” Explain how you'd decide whether that will help.

0 words