Skip to content
Episode 72 · Don't rewrite it in Go

Profiling: measure before you optimise

Wednesday, 10:05: after release 4.12, the hotel quote's p95 jumps from 250 ms to 2 s. The servers sit at 20% CPU, and Karim is already proposing a three-month rewrite in Go.

Before you speed anything up, find out where the time really goes. Your first guess is often wrong.

30 min intermediateMeasure before optimizingWork vs waiting: CPU-busy or blocked on I/OReading CPU profiles and flame graphsFinding the slow hop with distributed tracesLoad testing to find the breaking pointFinding the bottleneck
  1. Problem

    After a release, the hotel booking API's p95 latency jumps from 250 ms to 2 s. The team's first instinct: "Node is slow, let's rewrite the hot endpoint in Go."

  2. Why the simple system fails

    Guessing optimises whatever is easiest to see. Here CPU sits at 20%: the servers aren't working hard, they are waiting. A faster language would change nothing.

  3. Engineering concept

    Measure first. Decide whether time is spent working (CPU) or waiting (I/O, locks, queues), then zoom in with the tool that answers that question.

  4. Technology

    APM and distributed traces to find the slow hop, CPU profilers and flame graphs for compute, EXPLAIN ANALYZE and query statistics for the database, load tests to reproduce problems at scale.

  5. Trade-offs

    Every measurement has overhead and blind spots: sampling misses rare events, staging lies about production data, instrumenting everything costs money. Measure enough to decide, then stop.

01

The problem

2 min
Wednesday, 10:05. The morning after release 4.12.
  1. Omar · On-call SRE
    GET /hotels/:id/quote: p95 from 250 ms to 2,000 ms since 4.12. Pod CPU is 20%.
  2. Karim · CTO
    Node is too slow for this. Let's rewrite the quote endpoint in Go. About three months.
  3. Lina · Senior engineer, your mentor
    You lead this one; I'll only ask questions. Your first guess?
02

The big idea

4 min
Working or waiting?
The big idea

Measure before you optimise: is it working or waiting? Then find the biggest slice, and use the tool that looks inside it.

Karim's Go rewrite only shrinks our 230 ms of code: 2,010 ms becomes about 1,825. Three months for 9%. The index took ten minutes.

03

How it works

5 min
From symptom to cause
  1. 1
    Pin the symptom

    Which endpoint, which percentile, since when? Quote p95, since the 4.12 deploy at 21:30.

  2. 2
    Working or waiting?

    CPU low → waiting: trace it. CPU near 100% → working: profile the code. Here, 20%: waiting.

  3. 3
    Open a slow trace

    A trace is one request's timeline, split into spans, one per step. A 2,010 ms quote: auth 8, availability query 1,712, pricing code 230, render 60.

  4. 4
    Look inside the biggest span

    EXPLAIN ANALYZE runs a query and shows how it was executed. Here: a full scan, 4 million rows read to return 58. 4.12 added it with no index.

  5. 5
    If it's working: a flame graph

    A CPU profiler samples which functions run. Its flame graph draws them: the wider a box, the more CPU it used. Height is only call depth.

  6. 6
    Fix and re-measure

    Index on (hotel_id, night): the query takes ~4 ms. p95 back to ~240 ms, same traffic, same percentile.

04

Watch a decision being made

4 min
Worked example

You profile search ranking

On Sunday evenings, search-ranking pods run at 85% CPU and p95 is 520 ms. Find where the CPU goes and fix it without adding pods.

  1. 1Confirm it's work, then profile under real load

    A trace shows 450 ms of 520 inside rankHotels, with no queries or calls. Record a 30 s profile on one production pod at peak.

    bash
    node --cpu-prof --cpu-prof-dir=/tmp/profiles dist/server.js

    Why: Lina: "profile where the problem lives. A function that's cheap once can dominate at 2,000 requests a second."

  2. 2Find the widest box

    formatPrice is 45% of CPU samples; under it, most width is new Intl.NumberFormat.

    Why: Width is CPU time. A tall, thin tower is cheap.

  3. 3Name the cause, fix the cause

    Each request formats ~400 prices and builds a new formatter for every one. Create one per currency and reuse it.

    Why: Not "JavaScript is slow": one expensive object created 400 times per request. A one-line fix beats any rewrite.

  4. 4Profile again

    CPU 85% → 48%, p95 520 → 310 ms. The new widest box is JSON serialisation, 18%.

    Why: Re-measuring proves the fix and shows the next slice; stop when you meet the goal.

The whole thing in one line: Working, so profile; widest box, so fix it; then profile again. CPU nearly halved with no new pods.
05

In real life

3 min
Always profiling, or profiling on demand?

Google: continuous profiling of the fleet

  • Has published work on profiling its whole fleet continuously at low overhead
  • Engineers can compare a service's profile with last week's
  • Needs storage, tooling and a team to run it

Worth it when small CPU savings multiply over a huge fleet.

Netflix: flame graphs when investigating

  • Its engineers popularised flame graphs in widely read public talks
  • Profiles recorded on demand when a service misbehaves
  • Cheap most of the time; the problem may be gone when you look

Worth it when traces and metrics tell you where to profile.

06

What goes wrong

3 min
Profiling on a laptop with a tiny database

Query plans depend on data size; 500 rows hide what 9 million reveal.

Instead

Test with production-sized data or statistics.

Optimising by intuition

"Node is slow" leads to rewrites that save 9% of a problem an index fixes.

Instead

Measure first: working or waiting, then the biggest slice.

Declaring victory without re-measuring

The fix may not have worked, or the bottleneck moved.

Instead

Same endpoint, same percentile, same load, before and after.

07

Trade-offs

3 min
Tracing policyMonthly costKeepsMisses
100% of traces~€4,200EverythingNothing, but mostly stores boring requests
5% + every error and slow trace~€350Every trace you'd open in an incidentRare-but-normal paths; needs a buffering collector
Friday, 11:00. The tracing bill arrives.
  1. Omar · On-call SRE
    I want 100% of traces. Wednesday, the one trace that mattered could have been dropped.
  2. Karim · CTO
    5% of 2,000 requests a second is still thousands of traces an hour.
  3. Karim · CTO
    Keep the rare ones. Don't keep the millions of boring ones.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • Nothing gets optimised before it's measured, least of all a ~3-month rewrite.
  • The evidence a performance change needs: a trace showing the dominant span, a query plan or flame graph for it, and a before/after at the same percentile.
  • What data tests run against: production-sized (9 million reviews), not a laptop with 500.
Tell Claude Code
Since release 4.12, GET /hotels/:id/quote has p95 of about 2,000 ms instead of 250 ms, with pods at 20% CPU. Don't change code yet. Pull three slow traces and tell me which span dominates and by how much. If it's a query, run EXPLAIN ANALYZE on staging (production-sized data) and show the plan. Propose the smallest fix, such as an index, with the migration and how long it locks the table. After I approve, apply it in staging and rerun the same load test, comparing p95 before and after.
Check before you accept
  • The diagnosis cites span durations from a trace, not "probably the database".
  • The plan ran on production-sized data: a Seq Scan before, an Index Scan after.
  • Amdahl check: how much of the 2,010 ms can the change remove at most?
Red flags
  • It starts optimising or rewriting before showing where the time goes.
  • It proposes a faster language while CPU is at 20%.
  • It reports an average improving and calls it solved.
09

Check yourself

2 min
Q1/3Debugging
A booking API takes 2 s per request. Its servers run at 20% CPU and are not memory-constrained. What is the most likely explanation?
How confident are you?

Next episode: hotel pages send 3,000 identical reads a second to PostgreSQL, and its CPU sits at 90%.

Finished?

Mark it complete to update your skill map.

Next: Caching strategies
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Reading a flame graph

You lead the second investigation, on search ranking. This time the CPU is busy, so the tool changes. Lina adds one step.

Worked example

You profile search ranking

On Sunday evenings, search-ranking pods run at 85% CPU and p95 is 520 ms. Find where the CPU goes and fix it without adding pods.

  1. 1Confirm it's work

    CPU 85% at peak, and a trace shows 450 ms of the 520 ms inside the rankHotels span with no child spans (no queries, no calls).

    Why: Busy CPU plus time inside our own code means working. A CPU profile will see it; a query plan won't.

  2. 2Profile under real load

    Record a profile on one production pod during peak for 30 s (a sampling profiler adds little overhead).

    bash
    node --cpu-prof --cpu-prof-dir=/tmp/profiles dist/server.js
# or attach to a running process with your platform's profiler

    Why: Lina: "profile where the problem lives. A function that's cheap for one request can dominate at 2,000 requests a second."

  3. 3Find the widest box

    In the flame graph, formatPrice is 45% of CPU samples. Beneath it, most width is in new Intl.NumberFormat.

    Why: Width is CPU time. Height is only call depth: a tall, thin tower is cheap.

  4. 4Name the cause

    Each ranking request formats ~400 prices and builds a new number formatter for every one. Building a formatter is expensive; using one is cheap.

    Why: The cause picks the fix. It's not "JavaScript is slow"; it's one expensive object created 400 times per request.

  5. 5Fix the cause

    Create one formatter per currency and reuse it (a small map). Ship behind no flag: it's a pure function with tests.

    Why: A one-line change to the widest box beats any rewrite or extra pods.

  6. 6Profile again

    Same load: CPU 85% → 48%, p95 520 → 310 ms. The new widest box is JSON serialisation of the response, 18%.

    Why: Re-measuring proves the fix and shows the next target. The biggest slice always moves.

The whole thing in one line: You confirmed the pods were working, profiled them under real load, fixed the widest box in the flame graph (an expensive formatter rebuilt 400 times per request), and profiled again: CPU dropped by almost half with no new pods.
QuestionToolWhat it can't tell you
Which endpoint got slower, at which percentile?Metrics / APM dashboardsWhy it got slower
Which hop of the request is slow?Distributed traces (spans)Which line of code burns CPU inside a span
Which functions use the CPU?Sampling CPU profiler, flame graphTime spent waiting on I/O or locks
Why is this query slow?EXPLAIN ANALYZE, slow-query log, pg_stat_statementsHow it behaves under concurrent load
Where does it break as traffic grows?Load test (k6, Gatling, Locust)Real production data skew, unless you replay it
Pick the tool by the question. Start broad (metrics, traces), then go narrow (profiles, query plans).
11

Interactive simulation

12

Visual diagram

Click a component

A booking API with APM tracing. Walk through a measurement-first investigation of a slow hotel quote.

13

Analogy

  • A slow hotel check-inFollowing one guest with a stopwatchProfiling a requestA trace
  • A slow hotel check-in15 of 20 minutes waiting for the key-card printerProfiling a requestTime spent waiting, not working
  • A slow hotel check-inThe receptionist is idle most of the timeProfiling a requestLow CPU, high latency
  • A slow hotel check-inFilming the receptionist's handsProfiling a requestA CPU profile or flame graph
  • A slow hotel check-inSending a tour bus of 60 guestsProfiling a requestA load test

Hiring a faster receptionist changes nothing when the printer is the problem. Where the analogy breaks: in software, the "printer" is often invisible until you measure, because waiting doesn't show up as busy.

14

Check your understanding

Q1/3Decision
Match each question to the tool that answers it best:
Which hop of this checkout request is slow?
Which functions burn CPU in the pricing service?
Why does this SQL query take 1.7 s?
At what traffic does checkout start timing out?
How confident are you?
15

Practice ladder

Step 1 · Trace

Which tool first?

Debugging Warm-up

For each situation, say whether the service is working or waiting, and which tool you'd open first.

  1. The payments API's p95 went from 300 ms to 1.4 s at 09:00. CPU 15%. The payment provider had a status-page incident at 08:55.
  2. The image-resize worker processes 40 images a minute instead of 120 after a library upgrade. CPU 100%.
  3. The owner dashboard is slow only for the 3 hotel groups with 300+ hotels. CPU normal.
  4. Search is fine all day, but every request is slow between 02:00 and 02:20. CPU normal on search pods.
Step 2 · Complete

Finish the investigation order

Debugging Practice

The booking confirmation page's p95 doubled after yesterday's deploy. Lina's investigation has two blank steps. Fill them in.

  1. Pin the symptom: confirmation page, p95 from 400 to 800 ms, since the 17:00 deploy. p50 unchanged.
  2. Working or waiting? CPU unchanged at 30%.
  3. ___
  4. The slowest span in slow traces is SELECT * FROM loyalty_points WHERE guest_id = …, new in this deploy, 380 ms.
  5. ___
  6. Fix, then re-measure p50, p95 and p99 under the same traffic.
Step 3 · Solve

The invoice worker

Debugging Practice

The invoice worker creates a PDF for each booking. Since Monday, the queue grows every evening. Data:

  • Each worker pod: CPU 97%.
  • A 30-second CPU profile on one pod shows: renderPdf 22%, loadFont 58% (called inside renderPdf for each invoice), zlib.deflate 12%, other 8%.
  • Monday's release added two new fonts for Japanese and Korean invoices.

Karim wants to double the worker pods. Explain what's happening and propose a fix, with an expected result.

Step 4 · Decide

Load test before Black Friday?

Decision Stretch

Black Friday is in four weeks. Last year StayHub's peak was 3× a normal Friday. Since then, search and checkout changed a lot (batched calls, new transaction logic). Options:

  • A: a load test at 4× normal Friday traffic against a production-sized staging copy. About 5 days of work for 2 engineers, plus ~€800 of temporary infrastructure.
  • B: no load test; add 50% more pods the week before and watch dashboards on the day.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

A colleague says: "The API is slow, so let's rewrite it in a faster language." Explain how you would decide whether that could help.

0 words