Episode 84 · The trace that stops halfway

OpenTelemetry: instrumentation, context propagation and sampling

Tuesday, 16:05: guest BK-7731 was charged but never got a confirmation, and the trace stops at booking with a gap marked "2.4 s, unknown". Of yesterday's 1,200 failed checkouts, only 120 have a trace at all.

A trace only helps if it follows the request everywhere. OpenTelemetry is the shared standard that makes that happen.

30 min advancedOpenTelemetry instrumentation: auto, manual, SDK and collectorTrace context propagation across services and queuesHead vs tail trace samplingFinding the slow hop with distributed tracesLogs, metrics and traces: what each one answers

Problem
Traces stop at service boundaries, each team uses a different vendor SDK, and the tracing bill grows faster than traffic.
Why the simple system fails
Without a shared standard, every service invents its own headers and formats, traces break at the first queue, switching vendors means re-instrumenting everything, and keeping 100% of traces is unaffordable.
Engineering concept
One vendor-neutral API for producing telemetry, a standard header for carrying trace context between processes, and a pipeline that can sample, enrich and route data before it reaches any backend.
Technology
OpenTelemetry API + SDK (auto-instrumentation for HTTP, DB and queue clients, manual spans), W3C `traceparent`, OTLP protocol, the OpenTelemetry Collector, head and tail sampling.
Trade-offs
Auto-instrumentation is fast but noisy; manual spans carry business meaning but cost effort. Head sampling is cheap but blind; tail sampling keeps the interesting traces but needs a stateful collector.

The problem

2 min

Tuesday, 16:05. Guest support is on the phone with BK-7731.

Maya · Product manager
Guest BK-7731 was charged but never got a confirmation. Can we see what happened?
Omar · On-call SRE
The trace ends at booking. After that, a gap: "2.4 s, unknown".
Omar · On-call SRE
And of yesterday's 1,200 failed checkouts, only 120 have a trace.

The big idea

4 min

Where the trace breaks

The big idea

A trace is only as complete as its weakest hop: pass the context across every boundary. And decide which traces to keep after they end, not before.

Once the context crossed the queue, BK-7731's trace showed the rest: the email worker's call to the provider timed out and the job was dropped after one retry.

How it works

5 min

From code to one complete trace

1
Start the SDK first
is an open standard to produce traces, metrics and logs once and send them anywhere. Its SDK loads before the app and sets service.name, the sampler and the exporter.
2
Auto-instrumentation
Spans for HTTP servers, clients and database queries, without code changes, but only for modules loaded after the SDK.
3
Manual spans
Business steps like payment.capture, with attributes like booking.id. Never emails or card data.
4
Propagate context
Context propagation: the trace id and parent span id travel in a header. HTTP gets it automatically; our homemade queue client doesn't.
5
Export to the collector
The is a separate process that receives telemetry, redacts it, samples it and forwards it to backends. Switching vendors means changing its config, not a dozen services.
6
Sample at the end
decides at the start: cheap, blind (10% kept ~120 of 1,200 failures). decides after the trace ends: keep every error and slow trace.

Watch a decision being made

4 min

Worked example

You fix the async half, Lina reviews

Make BK-7731-style failures traceable end to end, through the queue, and keep every failed checkout's trace.

1Start the SDK before the app
node --import ./instrumentation.js worker.js.
Why: Lina: "modules imported before the SDK starts are never instrumented: silent gaps."

2Inject on publish, extract on consume

Booking puts the current context into the message headers; the worker continues from them.

propagation.inject(context.active(), headers); // adds traceparent
await queue.publish("booking.created", { bookingId }, { headers });
// in the worker:
const parent = propagation.extract(ROOT_CONTEXT, msg.headers);
await context.with(parent, () => handle(msg));

Why: Without the extract, the worker's spans get a new trace id: exactly the 2.4 s gap.

3Send everything to the collector
Every service exports 100% of spans to the collector, with a ParentBased sampler so no service cuts a trace in half.
Why: Tail sampling can only keep what it receives; head sampling would drop failures before anyone knew.
4Tail sample, routed by trace id
Keep all errors, all traces over 1 s, 5% of the rest; route spans by trace id so one collector sees each whole trace.
Why: ~48,000 traces a day instead of ~67,000, and all 1,200 failures kept instead of 120.

In real life

3 min

Two sampling philosophies

Google's Dapper: low-rate head sampling

The 2010 Dapper paper describes keeping a small uniform fraction, around 1 in 1,000
At Google's volume, still plenty of traces for typical latency
Cheap; most individual failures have no trace

Fits enormous volume and questions about the typical request.

Honeycomb: decide after the request ends

Publicly advocates tail-based and dynamic sampling, and publishes a sampling proxy
Keeps errors and slow requests, samples healthy traffic heavily
Needs buffering and routing a trace's spans to one place

Fits when you debug individual bad requests.

What goes wrong

3 min

Not propagating context through queues and jobs

Each consumer starts a new trace; the async half is invisible.

Instead

Inject into message headers, extract in the consumer.

Low-rate head sampling, expecting to debug errors

At 10%, 90% of failed requests have no trace.

Instead

Tail sampling: all errors and slow traces, a share of the rest.

Tail sampling on round-robin collectors

No replica sees a whole trace, so traces come out in fragments.

Instead

Route spans by trace id to one collector.

Trade-offs

3 min

Sampling	Failed checkouts traced	Costs
Head, 10% (today)	~120 of 1,200	~67,000 traces/day; nothing extra to run
Head, 20%	~240 of 1,200	Twice today's volume
Tail in the collector	All 1,200	~48,000 traces/day; collectors buffer and route

Friday, 09:30. Planning the collector rollout.

Omar · On-call SRE
Tail sampling means collectors buffering 10 s of every span. One more thing to run at 3 a.m. Keep head sampling, maybe at 20%.
Maya · Product manager
Support needs every failed checkout. At 20%, four out of five angry guests still get "we don't know".
Omar · On-call SRE
And if the collectors fall over, we lose all traces.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which traces you keep: every error, every checkout over 1 s, and 5% of the rest.
Which boundaries carry trace context: HTTP, and also the booking.created queue.
Which attributes a span may carry (booking_id yes, emails and card data never), and that vendor choices live in the collector config, not in app code.

Tell Claude Code

Traces for checkout stop at booking-service with a 2.4 s unknown gap, because our custom queue client doesn't carry trace context. Plan first. Using the OpenTelemetry propagation API, inject the current context into the message headers when publishing booking.created, and extract it in the payments and notifications workers so their spans continue the same trace. Don't replace the queue client and don't add a vendor SDK. Write a test that publishes inside a span and checks the worker's span has the same trace id. Then show me one staging trace from the gateway to the confirmation email.

Check before you accept

Open the staging trace: one trace id, gateway to email, no unexplained gap.
The test compares the trace id on both sides, not just that a span exists.
The SDK loads before express and pg; the collector rule keeps all errors (all 1,200 of yesterday's failures).

Red flags

It raises head sampling to 100% in every service, without a word on cost.
It installs a vendor's tracing library instead of the OpenTelemetry API.
It adds the guest's email as a span attribute "to find bookings faster".

Check yourself

2 min

Q1/3Debugging

In the tracing UI, POST /bookings ends after 120 ms, and a separate trace email.send_confirmation appears with a different trace id. What is missing?

How confident are you?

Next episode: Saturday night, checkout errors jump to 18%, five people join the call, and you run the incident.

Finished?

Mark it complete to update your skill map.

Next: Incident response: alerts, on-call, mitigation and blameless postmortems

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Keeping the traces that matter

Worked example

You fix the async half, Lina reviews

Make BK-7731-style failures traceable end to end, through the queue, and make sure every failed checkout keeps its trace.

1Start the SDK before the app
Run the worker with node --import ./instrumentation.js worker.js so HTTP and database modules are patched as they load.
Why: Lina's review: "Modules imported before the SDK starts are never instrumented, and you'd see silent gaps."

2Name the business step

Wrap the charge in a manual span with ids, not personal data.

tracer.startActiveSpan("payment.capture", async (span) => {
  span.setAttributes({ "booking.id": bookingId, "payment.provider": provider });
  try { return await capture(bookingId); }
  catch (err) { span.recordException(err as Error); span.setStatus({ code: SpanStatusCode.ERROR }); throw err; }
  finally { span.end(); }
});

Why: Auto spans say "POST /charges took 900 ms". A named span with booking.id says what the business was doing, and errors mark the trace for tail sampling.

3Inject context into the message
When booking publishes, put the current context into the message headers.
ts
```
const headers: Record<string, string> = {};
propagation.inject(context.active(), headers); // adds traceparent
await queue.publish("booking.created", { bookingId }, { headers });
```
Why: HTTP gets this for free; our homemade queue client doesn't. Every boundary needs the context.
4Extract it in the consumer
The worker continues the trace from the message headers instead of starting a new one.
ts
```
const parent = propagation.extract(ROOT_CONTEXT, msg.headers);
await context.with(parent, () => handle(msg));
```
Why: Without the extract, the worker's spans get a new trace id: exactly the 2.4 s gap.
5Send everything to the collector
Every service exports 100% of spans over OTLP to the collector, with a ParentBased sampler so no service cuts a trace in half.
Why: Tail sampling can only keep what it receives. Head sampling in each service would drop failures before anyone knew they failed.
6Tail sample in the collector
Keep all errors, all traces over 1 s, 5% of the rest. Route spans by trace id so one collector sees each whole trace.
Why: ~48,000 traces a day instead of ~67,000, and all 1,200 failures kept instead of 120.

Choice	Option A	Option B
Instrumentation	Auto: minutes to set up, generic span names	Manual: business names and attributes, takes engineering time
Sampling	Head: decided at start, cheap, blind to outcome	Tail: decided at end, keeps errors/slow, needs buffering
Export path	SDK → backend directly: simplest	SDK → collector → backend(s): redaction, sampling, vendor switching
Async link	Child span: the consumer is part of the request's tree	Span link: separate trace, linked (better for batches and long delays)

Interactive simulation

Visual diagram

Web app

Booking API (OTel SDK)

Inventory

booking.created queue

Email worker

OTel Collector

Tracing backend

Click a component

Context flows with the request; telemetry flows to a collector, which samples and exports.

Analogy

Parcel trackingOpenTelemetry

Parcel trackingTracking number on the parcelOpenTelemetryThe trace id
Parcel trackingA label format all carriers readOpenTelemetryThe standard trace header
Parcel trackingEach scan with a start and end timeOpenTelemetryA span
Parcel trackingThe sorting centreOpenTelemetryThe OTel Collector
Parcel trackingFull photos only for lost or late parcelsOpenTelemetryKeeping errors and slow traces

Where the analogy breaks: a parcel follows one path. One request can split into many parallel calls, so a trace is a tree of steps, not a single line.

Check your understanding

Q1/3Recall

Match each OpenTelemetry piece to its job:

API

SDK

Auto-instrumentation

Collector

traceparent header

How confident are you?

Practice ladder

Step 1 · Trace

Predict the waterfall

Debugging Warm-up

A cancellation flows like this:

The gateway receives POST /bookings/BK-8102/cancel (auto-instrumented HTTP).
It calls the booking service over HTTP (auto-instrumented client and server).
Booking publishes booking.cancelled on the queue with propagation.inject.
The refunds worker consumes it and calls propagation.extract.
The email worker also consumes it, but doesn't extract the context.

Trace it: how many traces does this cancellation produce, and which spans appear in each?

Step 2 · Complete

Fill in the propagation code

Code Practice

Lina's review left two gaps in the reviews service, which publishes a review.posted message consumed by a moderation worker.

// Producer (reviews API)
const headers: Record<string, string> = {};
/* gap 1 */
await queue.publish("review.posted", { reviewId }, { headers });

// Consumer (moderation worker)
queue.subscribe("review.posted", async (msg) => {
  /* gap 2 */
    tracer.startActiveSpan("review.moderate", async (span) => {
      try { await moderate(msg.body.reviewId); }
      finally { span.end(); }
    }),
  );
});

Step 3 · Solve

Design sampling for search

Estimation Practice

The search service handles 1,000,000 requests an hour. 0.5% fail and about 1% take longer than 1 s. Each trace has about 20 spans. Today: 2% head sampling.

How many failed searches per hour have a trace today?
Design a tail-sampling policy and estimate how many traces (and spans) per hour it keeps.
What must be true of the collectors for it to work?

Step 4 · Decide

Auto-instrument everything, or name the checkout steps first?

Decision Stretch

You have two weeks of one engineer's time for tracing. Options:

A. Auto-instrument all 12 services (HTTP, database, Redis), plus fix context through the 3 queues. Generic span names like POST /charges and SELECT room_availability.
B. Only the 6 checkout services, but with manual spans and attributes for the business steps (pricing.compute, inventory.reserve, payment.capture, email.send_confirmation) plus the queue fix.

Most incidents in the last quarter touched checkout; two touched search. Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain how one booking request becomes a single trace across an API, a downstream service and a queue worker, and how you would keep tracing affordable.

0 words