Episode 75 · Bigger, or more?

Scaling up vs scaling out

Tuesday, 10:00: StayHub's first TV ad airs on Sunday at 20:00, and marketing expects traffic to triple within 30 seconds. Karim asks the room: bigger servers, or more of them?

Your server is full. Buy a bigger one, or add more? Here's how to choose, and why the database plays by different rules.

30 min intermediateVertical vs horizontal scalingCost curves of scaling up vs outWhy stateful pieces resist horizontal scalingAutoscalingStateless serversFinding the bottleneck

Problem
A hotel booking API runs on one 4-core server. Traffic doubles every six months, and every Friday evening the server hits 100% CPU and requests start timing out.
Why the simple system fails
One machine has a ceiling: you can buy a bigger one a few times, each step costs more per core and needs a restart, and it is still a single point of failure.
Engineering concept
Scale up (vertical) adds resources to one machine; scale out (horizontal) adds more machines behind a load balancer. Stateless tiers scale out easily; stateful tiers usually scale up first, then need replication or sharding.
Technology
Larger instance types, load balancers, stateless app servers, autoscaling groups and the Kubernetes Horizontal Pod Autoscaler, read replicas and sharding for data.
Trade-offs
Scaling up is simple but capped, pricey at the top and a single point of failure. Scaling out is near-linear and resilient but needs statelessness, a load balancer and distributed-systems care. The database decides most real limits.

The problem

2 min

Tuesday, 10:00. StayHub is at about 1.2 million guests a day.

Maya · Product manager
Our first TV ad airs Sunday at 20:00. Marketing expects traffic to triple within 30 seconds.
Karim · CTO
So: bigger servers or more servers? I want one answer by Thursday.
Lina · Senior engineer, your mentor
Before I answer, make your own guess.

The big idea

4 min

Two ways to get more capacity

The big idea

Scale out what's stateless. Scale up what holds data, until you must copy or split it. Always ask which part runs out first.

My answer to Karim: more booking pods, a bigger database, and a pooler between them. The bigger database buys this Sunday, not forever.

How it works

5 min

Where Sunday's 3× lands

1
Load balancer
Spreads requests over however many pods exist. Scaling out () means adding machines that share the work behind it.
2
booking-api pods
: they keep nothing between requests, so any copy serves any guest. 11 pods → 31. Scale out.
3
Connection pooler
31 pods each opening a pool would push Postgres past its connection limit. keeps real connections at ~300 however many pods run.
4
PostgreSQL primary
One machine holds every booking: 32 vCPU at ~55% on a normal Sunday, so 3× won't fit. Scaling up () means a bigger machine: 64 vCPU.
5
The catch
64 vCPU costs ~€8,000/month vs ~€3,800: more than double, and the sizes above are steeper. Copies and splits come next.
6
Headroom and timing
Headroom is spare capacity kept free: plan pods at 70% CPU (175 req/s each). A new pod needs ~40 s, a new node ~2 min. The spike takes 30 s.

Watch a decision being made

4 min

Worked example

Lina plans capacity for the TV ad

Traffic will triple within 30 s on Sunday at 20:00 (1,800 → 5,400 req/s). Keep booking working without over-spending the rest of the week.

1Measure one unit, size with headroom
Load test one pod: ~250 req/s at 100% CPU. At 70%: 175 req/s. Ad: 5,400 ÷ 175 → 31 pods, plus one spare: 32.
Why: Capacity starts from what one unit really does, measured. The 30% left free absorbs bursts and a pod failing.
2Check the timing
Pod ~40 s, node ~2 min, spike ~30 s. Autoscaling alone always arrives late.
Why: Capacity that arrives after the spike is no capacity at all.
3Pre-scale and cap
Scheduled job at 19:45: 32 pods, 4 extra nodes. Autoscaler min 32 until 22:00, max 40.
yaml
```
spec:
  minReplicas: 32   # until 22:00, set by a scheduled job
  maxReplicas: 40   # protects the database
```
Why: The maximum protects the shared database from a bug or a bot scaling pods into a connection storm.
4Grow the shared part, then rehearse
PgBouncer in front; Postgres 32 → 64 vCPU on Saturday at 02:00 (~1 min failover). Saturday 10:00: 5,400 req/s for 15 min: p95 210 ms, database CPU 62%.
Why: Scale up the one part that can't be multiplied, at the quietest hour. A plan you haven't tested is a guess.

In real life

3 min

One very big machine, or many smaller ones?

Let's Encrypt: scale the database up

Has publicly described running its main database on a few very large servers
One primary with replicas; no splitting of data
Simple, but bounded by the biggest machine

Worth it while the data fits one machine and simplicity matters most.

Discord: scale message storage out

Has publicly described storing messages across a cluster of many database nodes
Trillions of messages: no single machine could hold them
Needs careful data modelling and hot-partition handling

Worth it when data and writes outgrow any single machine.

What goes wrong

3 min

Trusting autoscaling with a sudden, known spike

Pods take tens of seconds, nodes minutes; the spike arrives first.

Instead

Schedule capacity before known events; keep headroom and spare nodes.

Autoscaling with no maximum

A bug or retry storm scales pods until the database (or the bill) collapses.

Instead

Set maxReplicas from what shared dependencies can take.

Scaling out a tier that keeps state in memory

Guests land on pods that don't have their session or cart.

Instead

Make the tier stateless first.

Trade-offs

3 min

	Scale up	Scale out
Code changes	None	App must be stateless
Ceiling	Largest machine; price per core climbs	Shared dependencies
Failure	Single point of failure	Survives losing instances
Speed	Minutes to resize	Instances added automatically

Thursday, 14:00. Planning beyond Sunday.

Karim · CTO
Keep the 64-vCPU database. Next year, go to 96. No code changes, no new failure modes.
Lina · Senior engineer, your mentor
96 costs about €12,500 a month, then there's nothing bigger. And 94% of our queries are reads: copies take them off the primary for less.
Karim · CTO
Replicas mean lag bugs and more things to run. I've seen those go wrong.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which parts scale out (stateless booking-api pods) and which scale up (the PostgreSQL primary).
The CPU target and limits: 70% means 175 req/s per pod; the max (40) is a cost and safety ceiling you choose.
Whether a known spike gets scheduled pre-scaling (32 pods at 19:45) instead of trusting .

Tell Claude Code

We expect 3x traffic on booking-api at Sunday 20:00 (1,800 to 5,400 req/s). One pod handles about 175 req/s at our 70% CPU target. Plan first, don't change anything yet: propose a scheduled pre-scale to 32 pods at 19:45 plus 4 extra nodes, and an HPA max of 40. Don't touch the PostgreSQL size or the PgBouncer pool; I handle the database separately. Show me the files you'll change, the maths behind the pod count, and how to roll it back on Monday morning.

Check before you accept

Redo the maths: 5,400 ÷ 175 ≈ 31 pods.
Where the scale-down happens: a pre-scale with no end keeps paying for 32 pods all week.
Real database connections stay around 300 through PgBouncer, whatever the pod count.

Red flags

It raises the max to 100 "to be safe" without saying what that does to the database.
It adds pods to fix a database at 95% CPU.
It lowers the CPU target to 30% every day to absorb one Sunday spike.

Check yourself

2 min

Q1/3Reasoning

Each API instance handles 250 req/s at 100% CPU. You target 70% CPU for headroom. How many instances do you need for a 1,800 req/s peak?

instances

How confident are you?

Next episode: Sunday went well, but the morning after, 1,900 guests lost the room they were holding and some got the same email eleven times.

Finished?

Mark it complete to update your skill map.

Next: Stateless services

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Sizing for 20:00 on Sunday

Lina sizes the Sunday plan, thinking out loud. Every number comes from a measurement, not a guess.

Worked example

Lina plans capacity for the TV ad

Traffic will triple within 30 s on Sunday at 20:00 (1,800 → 5,400 req/s). Keep booking working without over-spending the rest of the week.

1Measure one unit
Load test one booking-api pod: ~250 req/s at 100% CPU, with p95 still under target up to ~70%.
Why: Capacity planning starts from what one unit really does, measured, not from the instance's spec sheet.
2Size with headroom
Plan at 70%: 175 req/s per pod. Normal Sunday: 1,800 ÷ 175 → 11 pods. Ad: 5,400 ÷ 175 → 31 pods, plus one spare: 32.
Why: The 30% left free absorbs bursts and one pod failing while others start.
3Check the timing
New pod: ~40 s to be ready. New node: ~2 min more. The spike: ~30 s. Autoscaling alone will always arrive late.
Why: Capacity that arrives after the spike is no capacity at all.

4Pre-scale and cap

Scheduled job at 19:45: 32 pods, 4 extra nodes. Autoscaler min 32 until 22:00, max 40.

yaml

spec:
  minReplicas: 32   # until 22:00, set by a scheduled job
  maxReplicas: 40   # protects the database
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

Why: The maximum protects the shared database; without it, a bug or a bot could scale pods into a connection storm.

5Protect and grow the shared part
PgBouncer in front of Postgres (~300 real connections whatever the pod count). Resize Postgres 32 → 64 vCPU on Saturday at 02:00 (~1 min failover).
Why: The database is the one part that can't be multiplied this week. Scale it up, at the quietest hour, and bound what reaches it.
6Rehearse
Saturday 10:00: load test at 5,400 req/s for 15 minutes on the real setup. p95 210 ms, Postgres CPU 62%.
Why: A plan you haven't tested is a guess. The rehearsal also proves the scheduled job and the node pool actually work.

Dimension	Scale up (vertical)	Scale out (horizontal)
Code changes	None	App must be stateless; shared state moves to external stores
Ceiling	Largest machine available; price per core rises at the top	Practically none for stateless tiers; limited by shared dependencies
Failure	Single point of failure; resize often needs a restart	Survives losing instances; rolling deploys without downtime
Elasticity	Slow: minutes to resize	Fast: add or remove instances automatically
Good fit	Databases, caches, early-stage everything	Stateless web/API tiers, workers, read replicas

Interactive simulation

Visual diagram

Guests

Load balancer

API instance 1

API instance 2

API instance 3 (autoscaled)

Autoscaler

Redis

PostgreSQL primary (16 → 64 cores)

Read replica

Click a component

One booking platform after scaling out its stateless tier. Walk through where each strategy applies.

Analogy

Getting guests to the hotelScaling

Getting guests to the hotelSwap the minibus for a big coachScalingScale up: a bigger machine
Getting guests to the hotelNo 500-seat coach existsScalingThe ceiling of one machine
Getting guests to the hotelRun several minibusesScalingScale out: more machines
Getting guests to the hotelA dispatcher assigns busesScalingA load balancer
Getting guests to the hotelNo bus keeps a guest's luggageScalingStateless servers
Getting guests to the hotelEveryone still queues at one receptionScalingThe single database

Where the analogy breaks: you can add a minibus in minutes once you own one. New servers take time to start, so autoscaling always reacts a little late. Keep some spare room.

Check your understanding

Q1/2Decision

Match each component to its usual first scaling move:

Stateless REST API

PostgreSQL primary taking all writes

Read-heavy search queries on that database

Queue workers processing image resizes

How confident are you?

Practice ladder

Step 1 · Trace

Trace a newsletter spike

Estimation Warm-up

search-api pods each handle 400 req/s at 100% CPU; you plan at 70%. Normal Tuesday peak: 2,100 req/s. A newsletter to 3 million guests goes out at 10:00 and doubles search traffic within 2 minutes. Pods take ~40 s to be ready; there's spare node capacity for 10 more pods.

How many pods for the normal peak, and for the newsletter?
With only CPU autoscaling (checks every 15 s), will search keep up? Explain.
What would you change?

Step 2 · Complete

Up or out?

Decision Practice

Complete Lina's table: for each component, the usual first scaling move and one reason.

Component	First move	Reason
booking-api (stateless pods)	___	___
Email worker (reads from a queue)	___	___
Redis cache (one node, 60% memory)	___	___
PostgreSQL primary	___	___
Search engine (OpenSearch, read-heavy)	___	___

Step 3 · Solve

Twelve pods, worse latency

Debugging Practice

The owner-portal API went from 6 to 12 pods for a busy Monday. p95 got worse (300 → 900 ms). Data:

Owner-portal pods: CPU 30%.
Postgres: CPU 95%, active connections 240 (were 120).
Each pod has a pool of 20 connections; no PgBouncer.
Monday morning, owners all load the same "weekly report" query (~400 ms, a large aggregation).

Explain what happened and propose fixes in order.

Step 4 · Decide

Pre-scale every weekend?

Decision Stretch

After the TV ad, Maya plans a campaign every Sunday at 20:00 for 8 weeks. Options:

A: a scheduled job pre-scales to 32 pods and 4 extra nodes from 19:45 to 22:00 every Sunday. Extra cost ~€90 per Sunday.
B: raise the autoscaler's minimum to 32 pods all week. Extra cost ~€4,500 a month.
C: rely on CPU autoscaling with a 50% target instead of 70%.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain when you would buy a bigger server and when you would add more servers.

0 words