Episode 83 · One guest in fifty

Logs, metrics and traces: the three pillars of observability

Monday, 11:20: one guest in fifty says checkout hangs for more than 4 seconds, yet the dashboard shows a 250 ms average and CPU at 30%. Six services touch checkout, and every team says theirs is fine.

When production breaks, you need clues. Metrics, traces and logs are three kinds of clues, and they work best together.

30 min intermediateLogs, metrics and traces: what each one answersMetric cardinality and label designRED and USE methods for dashboardsFinding the slow hop with distributed tracesLogging errors with context, never swallowing themp50 / p95 / p99 latency

Problem
Customers say checkout is slow. Twelve services are involved. You need to know what is wrong, where, and since when, in minutes.
Why the simple system fails
`console.log` strings on 40 servers can't be searched or aggregated, CPU graphs say nothing about users, and no single service's logs show where a request spent its time.
Engineering concept
Three complementary signals: metrics tell you *that* something is wrong (cheap, aggregated), traces tell you *where* (one request across services), logs tell you *why* (detailed events). Correlate them with a shared trace id.
Technology
Structured JSON logs (ELK / Loki), Prometheus-style metrics with Grafana dashboards, distributed tracing (Jaeger, Tempo), error trackers like Sentry, all-in-one platforms like Datadog.
Trade-offs
More detail costs more money and noise. Logs are expensive per event, metrics break with high-cardinality labels, traces must be sampled. Choose the cheapest signal that answers each question.

The problem

2 min

Monday, 11:20. A checkout now passes through six services.

Maya · Product manager
One guest in fifty says checkout hangs for 4 seconds or more. Some give up.
Omar · On-call SRE
Average latency 250 ms. CPU 30% on every service. Errors flat.
You · Engineer at StayHub
Every team says their service is fine. Where do we even start?

The big idea

4 min

Zooming in on the slow 2%

The big idea

Metrics say that. Traces say where. Logs say why. The trace id links them, so you can zoom from the whole fleet to one line.

The index came back at 12:05: the span dropped to about 12 ms and checkout p99 to about 350 ms. None of the three signals alone would have been enough.

How it works

5 min

From "something's wrong" to the line that causes it

1
Metrics: that
A metric is a number aggregated over time, cheap and good for alerts. Checkout p99 (the latency 99% stay under) is 4,200 ms while p50 is 180. The slow tail is real: 2% of guests.
2
Pick an example
Open one slow checkout as a trace. Its trace id is 7f3a…
3
Trace: where
A follows one request across services; each timed step is a span. Pricing 20 ms, payments 200 ms, inventory.reserve 3,900 ms.
4
Logs: why
A is an event written as named fields. Logs with trace_id 7f3a… show a full table scan on room_availability for a hotel with 340 rooms.
5
Fix, then confirm
Thursday's migration dropped an index. Recreate it, then check the p99 metric falls.
6
Two dashboard methods
per service: rate, errors, duration. per resource: utilisation, saturation, errors. RED says which service hurts; USE says which resource is exhausted.

Watch a decision being made

4 min

Worked example

Lina makes checkout answerable

Monday's diagnosis worked because we got lucky with one log line. Make the next checkout problem answerable with queries, without blowing up the telemetry bill.

1RED with percentiles, bounded labels
Each service: rate, errors, p50/p95/p99 from a histogram. Labels: route template, method, status class. 25 × 4 × 5 × 8 buckets = 4,000 series. No user, hotel or booking ids.
Why: Every label combination is a stored series; one unbounded label multiplies the count by millions.

2Log events, not sentences

JSON with an event name and consistent fields.

log.info(
  { event: "room.reserved", trace_id: span.spanContext().traceId,
    booking_id: bookingId, hotel_id: hotelId, duration_ms: ms },
  "room reserved",
);

Why: "All slow reserves for hotel 812 today" becomes a query. In logs, ids cost bytes, not series.

3One trace id everywhere, linked
Every service passes the trace context on and puts trace_id on every log line. The p99 panel links to example traces; each trace links to its logs.
Why: Without it, six services' logs are six piles you can't join. With it, an hour of guessing becomes five minutes.
4Budget the volume
Debug logs off by default, traces sampled, a cardinality limit per service.
Why: Telemetry is a cost line: debug logs alone are ~60% of our log volume.

In real life

3 min

Two teams, two starting points

SoundCloud: metrics first

Created Prometheus, which is publicly documented as starting there
Numbers with labels, scraped every few seconds, for dashboards and alerts
Cheap and fast, but says nothing about one request's path

Fits when you need fleet-wide alerts and trends first.

Uber: tracing across many services

Created Jaeger, a distributed tracing system it has publicly described
Built to follow requests across a very large number of services
Needs every service to pass the trace context along

Fits when slowness crosses service boundaries.

What goes wrong

3 min

Ids, emails or raw URLs as metric labels

Each value creates new series; cardinality explodes.

Instead

Bounded labels; ids in logs and span attributes.

Free-text logs

You can't filter them, and every service formats ids differently.

Instead

JSON events with the same field names everywhere.

Debug logging everywhere in production

Cost explodes, useful lines drown, personal data leaks.

Instead

Info-level events, debug switched on briefly, sensitive fields redacted.

Trade-offs

3 min

Signal	Answers	Cost driver	Bad at
Metrics	Is it broken? How much?	Number of series	One request
Traces	Where did this request spend time?	Spans × sampling rate	Totals (sampled)
Logs	What exactly happened?	Bytes ingested	Aggregation at scale

Friday, 10:00. The telemetry bill: €18,000 a month.

Karim · CTO
Telemetry costs more than our database. Drop tracing, keep metrics and error logs.
Lina · Senior engineer, your mentor
We found Monday's bug with a trace. Without one, six teams would still be arguing.
Lina · Senior engineer, your mentor
Debug logs are 60% of the volume. Cut those, sample traces at 10%, keep all three pillars.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which incident questions you must answer (is checkout slow, where, why), and which signal answers each.
Which labels a metric may have: route, method, status class. Never user_id or booking_id.
What every log line carries (trace id, booking id as named fields, same names in all six services), and what telemetry you pay for (~€18,000/month today).

Tell Claude Code

Checkout has a p99 of 4.2 s while the average is 250 ms, and we can't connect a slow request across services. Plan first, then implement in inventory-service only. Structured JSON logs with trace_id, booking_id, hotel_id and duration_ms as fields. Request duration as a histogram with only route, method and status_class; ids go in logs and spans, never metric labels. A span around inventory.reserve with hotel_id as an attribute. Don't change other services' log levels. Show a log line from a local run, the series it creates, and a query finding all logs for one trace id.

Check before you accept

Count the metric's labels and multiply their values. Any id: reject.
One real log line: JSON, with the trace id, named like in other services.
One trace with the inventory.reserve span: ~12 ms with the index, ~3,900 ms means the problem is still there.

Red flags

It adds user_id or booking_id as a metric label "to make filtering easy".
It turns on debug logging everywhere to investigate one slow path.
It builds the alert on average latency.

Check yourself

2 min

Q1/3Reasoning

A counter has labels route (25 values), method (4) and status_class (5). A developer adds hotel_id (9,000 hotels). Roughly how many time series can the counter now produce?

series

How confident are you?

Next episode: the payments team's spans don't connect to anyone else's, and the slowest checkouts are exactly the ones sampling threw away.

Finished?

Mark it complete to update your skill map.

Next: OpenTelemetry: instrumentation, context propagation and sampling

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Signals that answer questions

Worked example

Lina makes checkout answerable

Today's diagnosis worked because we got lucky with one log line. Make sure the next checkout problem can be answered with queries, not guesses, without blowing up the telemetry bill.

1Chart RED with percentiles
For each service: Rate, Errors, Duration as p50/p95/p99 from a histogram.
promql
```
histogram_quantile(0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m])))
```
Why: RED is what guests feel. Percentiles show the slow tail that averages hide.
2Keep labels bounded
Labels: route template, method, status class. 25 × 4 × 5 × 8 buckets = 4,000 series. No user, hotel or booking ids.
Why: Every label value combination is a stored series. One unbounded label multiplies the count by millions.
3Log events, not sentences
Log JSON with an event name and consistent fields.
ts
```
log.info(
  { event: "room.reserved", trace_id: span.spanContext().traceId,
    booking_id: bookingId, hotel_id: hotelId, duration_ms: ms },
  "room reserved",
);
```
Why: "All slow reserves for hotel 812 today" becomes a query. In logs, high-cardinality fields are fine: they cost bytes, not series.
4Carry one trace id everywhere
Every service passes the trace context on, and puts trace_id on every log line.
Why: Without it, six services' logs are six piles you can't join.
5Link the signals
The p99 panel links to example traces (exemplars); each trace links to its logs.
Why: Moving between zoom levels in one click is what turns an hour of guessing into five minutes.
6Budget the volume
Debug logs off by default, traces sampled, a cardinality limit per service.
Why: Telemetry is a cost line. Debug logs alone are ~60% of our log volume.

Two dashboard methods keep you from drowning in graphs. for every service that handles requests (rate, errors, duration). for every resource (utilisation, saturation, errors): CPU, connection pools, queues, disks. RED tells you which service hurts; USE tells you which resource is exhausted underneath.

Interactive simulation

Visual diagram

Guest

API gateway

Checkout

Inventory

Postgres

Metrics (Prometheus)

Traces (Tempo/Jaeger)

Logs (Loki/ELK)

Click a component

Telemetry flows from services into three stores, linked by the trace id. Follow an investigation from alert to root cause.

Analogy

In a hospitalIn your system

In a hospitalHeart-rate monitor beeps when something driftsIn your systemMetrics and alerts
In a hospitalPatient's path: triage, waiting, radiologyIn your systemA trace through services
In a hospitalThe doctor's detailed notesIn your systemLogs
In a hospitalPatient id on every paperIn your systemTrace id linking everything

Where the analogy breaks: a hospital writes notes for every patient. Your system handles millions of requests, so you can't keep a full trace of each one. You choose which to keep.

Check your understanding

Q1/3Reasoning

Match each question to the cheapest signal that answers it:

Has checkout error rate gone up in the last hour?

Which service made this specific booking request slow?

What value did the price field have when booking 5512 failed?

Which exception started after release v2.14, affecting how many users?

How confident are you?

Practice ladder

Step 1 · Trace

Read a search trace

Debugging Warm-up

A guest's hotel search took 2,450 ms. Here's its trace (children are indented; they run inside their parent):

GET /search                     2450 ms   gateway
  search.query                  2400 ms   search
    pricing.batch_quote          310 ms   pricing
    reviews.summaries           2010 ms   reviews
      db.query reviews_avg      1980 ms   reviews
    photos.urls                   40 ms   photos

Pricing, reviews and photos are called one after another. Trace it: which span would you fix first, what would the search take if that span dropped to 30 ms, and what would you look at next to learn why it's slow?

Step 2 · Complete

Fix the payments log line and metric

Code Practice

The payments service has this log line and metric. Lina marked two gaps: rewrite the log line as a structured event (gap 1), and choose the metric labels (gap 2).

console.log(`Payment for booking ${bookingId} by ${email} took ${ms}ms, status ${status}`);

const paymentDuration = new Histogram({
  name: "payment_request_duration_seconds",
  labelNames: [/* gap 2 */],
  buckets: [0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});

Candidate labels: booking_id, email, provider (2 values), outcome (approved / declined / error), card_country (~190), hotel_id (9,000).

Step 3 · Solve

Confirmation emails that never arrive

Debugging Practice

Maya: "About 3 guests a day say they never got their confirmation email." Bookings succeed; the notifications service picks jobs from a queue and calls an email provider. Today it has only free-text logs and host CPU graphs.

Plan the investigation and the telemetry you'd add: which signal answers which question, and what exactly you'd emit.

Step 4 · Decide

Vendor or self-hosted telemetry?

Decision Stretch

StayHub pays an all-in-one telemetry vendor €18,000 a month. After cutting debug logs and sampling traces, the estimate drops to €11,000. Omar's alternative: self-host open-source metrics, logs and traces: about €5,000/month of infrastructure plus half an engineer (~€4,500/month) to run it. Everything is instrumented with OpenTelemetry already, so switching is possible.

The team is ~40 engineers; on-call relies heavily on the vendor's linked views (metric → trace → logs).

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a new teammate how logs, metrics and traces differ and how you'd use them together during an incident.

0 words