Episode 12 · The spinner that waited in line

Async and concurrency: doing many things while waiting

Tuesday, 11:30: hotel pages for the new chain take 2.4 s, while our own hotels load in 0.4 s. Our server's CPU sits at 6%, so it isn't working hard. It's waiting.

A server spends most of its life waiting. Async code lets it serve other people during that wait, and that's how one thread handles thousands of requests.

30 min foundationThe JavaScript event loopCallbacks, promises and async/awaitConcurrency vs parallelismRace conditions and check-then-act bugsProcesses vs threadsSynchronous vs asynchronous communication

Problem
A booking API must serve thousands of users at once, and every request spends most of its time waiting on the database, a payment provider or an email service.
Why the simple system fails
Synchronous code blocks while it waits: one slow call freezes the thread, so the server handles one request at a time. One thread per request runs out of memory long before you run out of users.
Engineering concept
Asynchronous I/O: start the wait, give the thread back, resume when the result arrives. Concurrency is juggling many tasks; parallelism is running them on many cores at once.
Technology
Callbacks, promises and async/await on top of the JavaScript event loop; threads, worker pools and processes for CPU-bound work; transactions and locks against race conditions.
Trade-offs
Async makes waiting cheap but not computation: one CPU-heavy task still freezes an event loop. Concurrency introduces interleavings, so shared state needs atomic operations or locks.

The problem

2 min

Tuesday, 11:30. The chain's first 300 hotels went live last week.

Maya · Product manager
Chain hotel pages take 2.4 s. Ours: 0.4 s. Guests leave before the rates show.
Omar · On-call SRE
CPU on the app server: 6%. It's not busy.
Lina · Senior engineer, your mentor
So it's waiting on something. What would you change?

The big idea

4 min

Waiting in line for no reason

The big idea

Async doesn't speed up a wait. It lets waits overlap.

The chain's API still takes 900 ms. What changes is what we do while waiting: start our other independent waits, and serve other guests.

How it works

5 min

Most of a request is I/O wait: time spent waiting on a database, another API, a disk. await () pauses this function until the result arrives, and gives the thread back. Inside one Node.js process:

What one Node.js process really does

1
Run a short burst
The handler checks the input and starts a call: microseconds of CPU.
2
Hand off the wait
await gives the network wait to the operating system and frees the one JavaScript thread.
3
Serve other guests
Meanwhile the thread runs other guests' handlers: , many tasks in progress on one thread. is several cores computing at once.
4
Answer arrives
The queues our function to resume right after the await, variables intact.
5
Overlap our own waits
Start the three calls together with Promise.all: we wait about 900 ms, the slowest, not 2.4 s.

Two limits. work (an 800 ms invoice PDF) holds the one thread, so every guest on that process waits: move it to a . And with many requests in progress, two can read and change the same data at once: a .

Watch a decision being made

4 min

Worked example

Fixing the 2.4-second chain hotel page

Get the page under a second without making any service faster, and without slowing any other page.

1Time each wait, draw the dependencies
Rates 900 ms, reviews 800 ms, room details 700 ms, rendering 5 ms. Each needs only the hotel id: none depends on another.
Why: Only independent calls can overlap. If reviews needed the rates, that order would stay.

2Start together, wait together

Start all three, then wait for all.

const [rates, reviews, rooms] = await Promise.all([
  chainApi.rates(id),     // 900 ms
  reviewsApi.get(id),     // 800 ms, overlapping
  db.roomDetails(id),     // 700 ms, overlapping
]);
return render(rates, reviews, rooms);

Why: The page pays for the slowest wait: about 900 ms.

3Decide what a failure means
Reviews fail? Show the page without them: Promise.allSettled for optional parts, and a timeout (say 2 s) on each call. Rates must succeed.
Why: Promise.all fails as soon as one call fails. A guest can book without reviews, not without a price.
4Measure again
0.91 s. CPU still about 6%, other pages' p95 unchanged.
Why: Proves the fix, and that the problem didn't move elsewhere.

In real life

3 min

Two ways to juggle many waiting requests

Node.js: one thread, an event loop

Its documentation describes one JavaScript thread with non-blocking I/O
Thousands of waiting requests cost little memory
One CPU-heavy task blocks every request on that process

Great when requests mostly wait.

Go: many lightweight threads

Each request runs in a goroutine; the runtime switches while it waits
Uses all cores in one process; shared data needs locks

Great when requests mix waiting with real computing.

What goes wrong

3 min

Thinking async makes code run in parallel

Only the waiting overlaps; computing still runs on one thread.

Instead

Async for I/O; workers or background jobs for CPU-heavy work.

Awaiting independent calls one by one

Waits add up: 900 + 800 + 700 ms instead of 900.

Instead

Start them together with Promise.all or allSettled.

Check-then-act on shared data

Between read and write, another request changes it: two guests get the last room.

Instead

One atomic step: a conditional UPDATE, a constraint, or a row lock.

Trade-offs

3 min

Model	Fits when	You pay	Main risk
Event loop (Node.js)	Mostly waiting on databases and APIs	One core per process	CPU work blocks everyone
Thread per request	Moderate traffic, blocking libraries	Memory per thread	Shared-memory races
Parallel workers / job queue	PDFs, images, data crunching	More parts to run	Failed jobs nobody sees

Monday, 10:00. Month-end invoices are coming again.

Karim · CTO
Keep the invoice PDF in the booking handler. Guests get it instantly, no queue to run.
Lina · Senior engineer, your mentor
Each PDF is 800 ms of CPU. Every other guest on that process waits 800 ms too. At month end, hundreds.
Karim · CTO
A worker means more jobs, failures and babysitting for six people, and the PDF arrives later by email.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which calls are truly independent and can run together, and which must wait for another's answer.
How many requests at once StayHub may send a partner: 50 per second per key for the chain, so a job over 36,000 rooms needs a limit.
What happens on a 429: growing pauses, a maximum number of tries and an alert, never instant retries.

Tell Claude Code

The chain hotel page awaits three independent calls in sequence: chain rates (900 ms), reviews (800 ms), room details (700 ms), 2.4 s total. Run them together so the page waits only for the slowest, and show what happens if one fails. Separately, the nightly rate refresh must never exceed 50 requests per second to the chain's API; on a 429, back off and retry at most 5 times, then log and alert. Don't change the booking flow. Plan first, then show the page time and a rate test.

Check before you accept

Claude says, call by call, whether it depends on another. Only independent ones start together.
The page measures about 0.9 s, and still shows rates and rooms when reviews fail.
The refresh test counts requests per second against 50, with backoff and a retry cap.

Red flags

36,000 calls in one Promise.all, called "faster".
429s retried immediately in a loop.
A setTimeout delay to "fix" a double booking: timing tricks make races rarer, not impossible.

Check yourself

2 min

Q1/3Reasoning

A hotel page needs room details (120 ms), reviews (200 ms) and prices (150 ms). The calls are independent. With await Promise.all([...]), about how long does the page wait, in ms?

How confident are you?

Next episode: the chain's API starts timing out at 02:00, and the refresh script dies on the first error, leaving nearly 35,000 rooms with stale prices.

Finished?

Mark it complete to update your skill map.

Next: Errors and exceptions: failing loudly, safely and usefully

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

One thread, many guests

You lead the timeline this time; I'll check. The rule we use: start everything that doesn't depend on something else, wait only where you must, keep CPU work off the event loop.

Worked example

Fixing the 2.4-second chain hotel page

The chain hotel page takes 2.4 s with the CPU at 6%. Get it under a second without making any service faster, and make sure no other page gets slower.

1Time each wait
Log how long each call takes: chain rates 900 ms, reviews 800 ms, room details 700 ms, our rendering 5 ms.
Why: Low CPU plus a slow page means waiting. Timing each wait shows which ones matter and that computing doesn't.
2Draw the dependencies
Does any call need another's answer? Rates need the hotel id (we have it). Reviews need the hotel id. Room details need the hotel id. None depends on another.
Why: Only independent calls can overlap. If reviews needed the rates, that order would have to stay.

3Start together, wait together

Start all three, then wait for all.

const [rates, reviews, rooms] = await Promise.all([
  chainApi.rates(id),     // 900 ms
  reviewsApi.get(id),     // 800 ms, overlapping
  db.roomDetails(id),     // 700 ms, overlapping
]);
return render(rates, reviews, rooms);

Why: The three waits now happen at the same time, so the page pays for the slowest one: about 900 ms.

4Decide what a failure means
If reviews fail, should the page fail? No: show the page without reviews. Use Promise.allSettled for optional parts, and give each call a timeout (say 2 s).
Why: Promise.all fails as soon as one call fails. A guest can book without reviews, but not without rates.
5Keep CPU off the loop
Check the handler for heavy computing. Rendering is 5 ms: fine. Invoice PDFs (800 ms) block the loop too: they need a worker of their own, a job we note for later.
Why: Overlapping waits helps one page; a blocked event loop hurts every page on the process. Both matter.
6Measure again
New timeline: 0.91 s. CPU unchanged at about 6%, and other pages' p95 unchanged.
Why: Measuring proves the fix and that we didn't move the problem somewhere else.

Race conditions need a different fix. Two app processes don't share memory, so the check and the write must happen together, in the database:

sql

-- Race-prone: check, then act, in two steps
--   SELECT available FROM room_inventory WHERE ...;   -- both requests see 1
--   UPDATE room_inventory SET available = available - 1 WHERE ...;

-- Atomic: the check and the write happen in one statement
UPDATE room_inventory
SET available = available - 1
WHERE room_type = 'deluxe' AND date = '2026-12-24' AND available > 0
RETURNING available;
-- 0 rows updated → sold out: tell the guest. No double booking.

The database lets only one request change that row at a time, so only one gets the last room.

Interactive simulation

Visual diagram

Guests' requests

Call stack (1 thread)

OS / libuv I/O

PostgreSQL

Payment provider

Microtask queue

Task queue

Click a component

Inside one Node.js process handling a booking request. Start the walkthrough.

Analogy

At hotel receptionIn code

At hotel receptionStaying on hold with the laundryIn codeBlocking (synchronous) code
At hotel reception"Call me back when it's ready"In codeA callback
At hotel receptionA sticky note promising an answerIn codeA promise
At hotel receptionFinish this, then take the next callbackIn codeThe event loop
At hotel receptionOne receptionist juggling 20 requestsIn codeConcurrency
At hotel receptionTwo receptionists giving out the same keyIn codeA race condition

Where the analogy breaks: hiring more receptionists is parallelism. A single JavaScript thread can't do that on its own. For heavy calculations, you hand the work to separate workers or background jobs.

Check your understanding

Q1/3Code

In what order are the letters printed?

console.log("A");
setTimeout(() => console.log("B"), 0);
Promise.resolve().then(() => console.log("C"));
console.log("D");

How confident are you?

Practice ladder

Step 1 · Trace

Trace two guests and the last room

Scenario Warm-up

The deluxe room for 24 December has available = 1. The booking code:

async function book(roomType: string, date: string) {
  const { available } = await db.one("SELECT available FROM room_inventory WHERE room_type=$1 AND date=$2", [roomType, date]);
  if (available < 1) return "sold out";
  await db.none("UPDATE room_inventory SET available = available - 1 WHERE room_type=$1 AND date=$2", [roomType, date]);
  await db.none("INSERT INTO bookings ...");
  return "booked";
}

Guest A clicks Book at 10:00:00.000, guest B at 10:00:00.010. Each database call takes about 20 ms. Trace both requests step by step. What does each guest see, and what is available at the end?

Step 2 · Complete

Finish the booking confirmation handler

Code Practice

After payment, the confirmation handler does five things:

chargeCard(quote): 300 ms
saveBooking(quote, payment): 20 ms (needs the payment id)
sendConfirmationEmail(booking): 150 ms (needs the booking)
notifyHotel(booking): 400 ms (needs the booking)
loadUpsells(quote.hotelId): 250 ms (needs only the hotel id)

Lina's skeleton has gaps. Fill them so the guest waits as little as possible, and state the total wait.

const upsellsP = ___;                 // start early?
const payment = await chargeCard(quote);
const booking = ___;
await ___;                            // email + hotel
const upsells = await upsellsP;

Step 3 · Solve

Search freezes every 30 seconds

Debugging Practice

Omar: "Every 30 seconds, search p95 jumps from 150 ms to 1.1 s for a moment, on every process. CPU spikes at the same moment."

You find this in the app code, running in every Node process:

setInterval(() => {
  const all = JSON.parse(fs.readFileSync("hotels-cache.json", "utf8")); // 40 MB file
  hotelsById = new Map(all.map((h) => [h.id, h]));
}, 30_000);

Explain what is happening and propose a fix.

Step 4 · Decide

How many at once?

Decision Stretch

The nightly chain-rates refresh must update 36,000 chain rooms (one call per room). Facts:

The chain's API allows 50 requests a second; each call takes about 900 ms.
One at a time, the refresh takes 36,000 × 0.9 s ≈ 9 hours: too slow.
Prices must be fresh by 06:00; the job starts at 02:00 (4 hours).

Karim suggests 200 at a time "to be safe on time". Choose a concurrency limit and write it as "We chose X over Y because…, accepting the risk of…". Show the arithmetic.

Explain it yourself

Explain how a single-threaded Node.js server can serve thousands of concurrent bookings, and the two situations where that stops working.

0 words