What is a server?
Friday, 20:04: Maya's flash-sale email reached 60,000 past guests four minutes ago. Traffic is five times the usual, search takes 8 seconds, and our one server in Frankfurt shows 100% CPU.
A server is just a program that waits for requests and answers them, running on a machine with limited power. Here's what that really means.
- Problem
Thousands of guests search for hotels at the same time, and each expects an answer in well under a second.
- Why the simple system fails
One program on one machine has a fixed amount of CPU time per second. Past that, requests wait in a queue, latency explodes, clients time out and retry, and things get worse.
- Engineering concept
A server is a process listening on a port, using CPU, memory, disk and network. Its capacity is finite; queueing makes latency non-linear near saturation. Stateless servers can be copied.
- Technology
Operating system processes and threads, web servers (Node, Gunicorn, Go net/http), reverse proxies (nginx), load balancers, autoscaling.
- Trade-offs
A bigger machine (vertical) is simple but capped and a single point of failure. More machines (horizontal) scale further but require statelessness and a load balancer.
The problem
2 min- Omar · On-call SRE225 requests per second. A normal Friday is 45. CPU at 100%. Search p99 went from 150 ms to 8 s.
- Karim · CTOFive times the traffic, fine. But fifty times slower? What is that server even doing?
- Lina · Senior engineer, your mentorGuess first, then we look inside it.
The big idea
4 minA server is fast until it's nearly full. Then requests queue and latency explodes.
The server isn't slower at each search tonight. It's full, and full means waiting. So you plan for headroom below the cliff, not for the edge.
How it works
5 minA is a program that waits on a port (a numbered door on a machine, 443 for HTTPS) and answers requests, plus the machine it runs on, with limited CPU, memory, disk and network.
- 1Knock on the port
The guest connects to our IP on 443. The operating system hands the connection to our app .
- 2Compute
A worker parses the request and filters hotels: about 20 ms of CPU on one core.
- 3Wait
It asks PostgreSQL for prices: about 60 ms of waiting. The core serves someone else meanwhile.
- 4Answer
It turns the result into JSON and sends the response.
- 5Or queue
If every core is busy, the request waits. Its time = wait in line + its own work.
- 6Watch utilisation
Utilisation is how busy a resource is, 0–100%. Past about 80%, waits grow very fast: at 90% a request takes about 10× its normal time.
The ceiling: 4 cores × (1,000 ms ÷ 20 ms) = about 200 searches per second. A normal Friday is 45 (CPU ~22%). Tonight 225 arrive, so the queue grows by 25 every second. Timeouts fire, the app retries, and retries join the same queue.
Watch a decision being made
4 minLina sizes the server for the next sale
Another flash sale in two weeks, same 225 searches per second at peak. Keep search under 200 ms without rewriting StayHub.
- 1Measure the cost of one request
Each search uses about 20 ms of CPU and waits about 60 ms on the database.
Why: CPU per request fills the cores. Waiting doesn't use a core, so it doesn't count toward the ceiling.
- 2Compute the ceiling, and size for 60%
4 cores × 50 = 200/s at 100% CPU, below 225. Target about 60% busy at peak: 225 ÷ 0.6 ≈ 375/s needed.
Why: Near 100% the wait in line dominates. At 60%, spikes stay in the flat part of the curve.
- 3Pick the cheapest fix that gets there
An 8-core machine gives about 400/s: 225 ÷ 400 ≈ 56% busy, with no code change. A second 4-core server would also work, but sessions live in memory, so it would log guests out.
Why: With two weeks to go, a change with no code is the safest one.
- 4Prove it before the sale
Load-test at 250/s and watch CPU and p99. Expect about 60% CPU and p99 well under 200 ms.
Why: The ceiling is an estimate. A load test finds the real knee before guests do.
In real life
3 minStack Overflow: a few powerful servers
- Its engineers have publicly described serving a huge site from a small number of large servers
- Few machines to run, debug and pay for
- Relies on tuned code and plenty of headroom per box
Works when traffic is steady and a small team wants few moving parts.
Netflix: many small, stateless servers
- Has publicly described running services as many interchangeable cloud instances
- Adds and removes instances as traffic rises and falls
- Needs load balancers, shared stores and automation
Works when traffic swings a lot and losing one machine must not matter.
What goes wrong
3 minIt stays flat, then goes vertical near saturation. A server at 70% CPU can be one email away from collapse.
Load-test to find the knee and keep headroom below it.
If requests wait on the database or an API, more servers just add more waiting requests.
Find what requests are waiting on before scaling anything.
Retries multiply traffic exactly when the server can least afford it.
Retry with and jitter, and cap the attempts.
Trade-offs
3 min| Option | Fits when | You pay | Main risk |
|---|---|---|---|
| Scale up (bigger machine) | Capacity needed this week, or the database | Steep prices at the top end | Single point of failure |
| Scale out (more machines) | Uptime matters, traffic swings | Load balancer, health checks, shared sessions | Bottleneck moves to the database |
| Headroom (~60% CPU) | Anything guests wait on | ~40% idle capacity | Paying for unused power |
- Karim · CTOLet's go straight to 4 small servers with autoscaling. No more single machine.
- Lina · Senior engineer, your mentorOne 8-core box handles the sale at 56% CPU. Four servers mean four times the things to watch.
- Karim · CTOAnd when your one box dies during the sale, StayHub is down.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- How much headroom you keep: a 225 req/s sale needs the 8-core box (~400 req/s) before the email goes out, not after.
- Bigger machine or more machines. A second server only works once sessions live outside the server's memory.
- Which numbers you watch during a sale (CPU, queue, p99, error rate) and at what level someone gets paged.
In two weeks marketing emails 60,000 guests about a flash sale. Last time traffic hit about 225 req/s, our 4-core server reached 100% CPU and p99 8 s. Each search costs about 20 ms CPU and 60 ms waiting on PostgreSQL. Plan the move to an 8-core server. Don't add a second server: sessions are in memory. Show the plan, expected CPU at 225 req/s and the rollback first. Then write a load test ramping to 250 req/s on staging, reporting CPU, p99 and errors.
- Do the arithmetic yourself: 8 cores × 50 = ~400 req/s, so 225 req/s is ~56% CPU.
- The load test reports p99, not only the average, which hides guests who wait 8 s.
- The rollback step is written down and doable in minutes.
- A second server behind a load balancer while sessions are still in memory.
- More servers when CPU is low and latency high: the time is spent waiting.
- Timeouts quietly raised so errors vanish from the graph while guests still wait.
Check yourself
2 minNext episode: two guests receive a confirmation for the same last room in Porto, and Lina asks why we trust the database to keep that from happening.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Under every framework, a server is a few lines: a asks the operating system for a port and runs a function for each request.
import { createServer } from "node:http";
const server = createServer(async (req, res) => {
// runs once per request
const hotels = await db.query("SELECT id, name FROM hotels WHERE city = $1", ["lisbon"]); // waits on the database
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify(hotels)); // CPU: build the JSON
});
server.listen(3000); // ask the OS: "send me every connection to port 3000"Lina sizes the server for the next sale
Maya plans another flash sale in two weeks and expects the same 225 searches per second at peak. Make sure search stays under 200 ms, without rewriting StayHub.
- 1Measure the cost of one request
From the server's metrics on a normal day: each search uses about 20 ms of CPU and waits about 60 ms on the database.
Why: CPU per request is what fills the cores. Waiting time doesn't use a core, so it doesn't count toward the ceiling.
- 2Compute the ceiling
4 cores × (1,000 ÷ 20) = 200 searches per second at 100% CPU.
Why: It turns "is the server big enough?" into a number you can compare with the forecast.
- 3Compare with the forecast at a safe level
225 is above the ceiling. And we don't want 100%: at 90% a request takes about 10× its normal time. Target about 60% busy at peak, so we need a ceiling of about 225 ÷ 0.6 ≈ 375 per second.
Why: Near 100% the wait in line dominates. Sizing for 60% keeps spikes in the flat part of the curve.
- 4Pick the cheapest fix that gets there
An 8-core machine gives about 400 per second: 225 ÷ 400 ≈ 56% busy. It needs no code change. A second 4-core server would also work, but our sessions live in memory, so it would log guests out.
Why: With two weeks to go, a change with no code to write is the safest one.
- 5Prove it before the sale
Run a load test at 250 per second against the new machine and watch CPU and p99. Expect about 60% CPU and a p99 well under 200 ms.
Why: The ceiling is an estimate. A load test finds the real knee of the curve before guests do.
| Utilisation | Latency multiplier (≈ 1 / (1 − ρ)) | A 50 ms request takes… |
|---|---|---|
| 50% | 2× | ~100 ms |
| 70% | 3.3× | ~170 ms |
| 80% | 5× | ~250 ms |
| 90% | 10× | ~500 ms |
| 95% | 20× | ~1 s |
| 99% | 100× | ~5 s, then timeouts |
Interactive simulation
Visual diagram
One request's life inside a server, and what happens when you need more than one.
Analogy
- At a hotel reception"Reception, ground floor"In a serverThe port where requests arrive
- At a hotel receptionReceptionists working side by sideIn a serverThreads sharing one process
- At a hotel receptionHow fast they workIn a serverThe CPU
- At a hotel receptionDesk space and the archive roomIn a serverMemory and disk
- At a hotel receptionA line out the doorIn a serverRequests queueing when the server is full
- At a hotel receptionOpening a second identical deskIn a serverAdding a stateless server
Where the analogy breaks: a receptionist helps one guest at a time. One server thread can juggle thousands of requests, as long as most of them are just waiting for an answer from somewhere else.
Check your understanding
Practice ladder
Trace the hotel-page load
Estimation Warm-upStayHub's 8-core server now also serves hotel pages. Each hotel page uses 40 ms of CPU and waits 30 ms on the database.
On Sunday evening, 120 hotel pages per second arrive (and nothing else).
- What is the ceiling for hotel pages per second on 8 cores?
- How busy is the CPU at 120 per second?
- Using the table from this lesson, roughly how long does a hotel page take compared with when the server is idle?
Finish Lina's sizing note for the summer peak
Estimation PracticeMaya expects the summer peak to reach 300 searches per second. Each search uses 20 ms of CPU. Lina's sizing note has two gaps:
- Measure the cost of one request: 20 ms of CPU per search.
- Compute the ceiling per core: 1,000 ÷ 20 = 50 searches per second per core.
- Compare with the forecast at a safe level: ___
- Pick the cheapest fix that gets there: ___
- Prove it: load-test at a bit above 300 per second and watch CPU and p99.
Fill in steps 3 and 4. Our servers are now stateless (sessions are in Redis), and machines come with 8 or 16 cores.
Slow owner reports, idle CPU
Debugging PracticeHotel owners like Nadia download a monthly report from /owner/reports. On the 1st of the month, Omar sees:
- Report p99: 14 s (normally 900 ms)
- App server CPU: 18%
- Database CPU: 95%
- Each report runs one big query that takes about 800 ms on a quiet database
Karim wants to add two more app servers before tomorrow. Explain what's really happening and what you'd do instead.
Downsize to save money?
Decision StretchKarim looked at the cloud bill. "Our 8-core server averages 5% CPU. Let's go back to 4 cores and save €180 a month."
The facts: - Average load: 20 searches per second. Normal Friday peak: 45 per second. Sale nights: up to 225. - Each search uses 20 ms of CPU. - Maya plans one sale a month. One sale night brings about €9,000 of bookings.
Write a short decision: "We chose X over Y because…, accepting the risk of…"
Explain it yourself
Explain what a server is, and why a server at 95% CPU is much slower than one at 60%, even though it's only doing ~1.5× more work.