Episode 139 · The run nobody could explain

Agent traces: seeing what the AI actually did

Monday, 9:05. A guest says the booking agent took 14 seconds and booked 15 November instead of the 5th. Finance asks why September's model bill doubled to 38,000 €. Our logs say `POST /chat 200 14.2s`.

When an AI agent does something strange, a trace shows you exactly what it saw, did and cost, step by step.

30 min advancedAnatomy of an agent trace: model, retrieval and tool spansTracing and monitoring agentsToken usage and cost attribution per traceAttaching eval scores and feedback to production tracesPrivacy of prompt and trace logs: redaction, retention, accessOpenTelemetry instrumentation: auto, manual, SDK and collector

Problem
A guest complains the booking agent “took forever and then booked the wrong dates”. Finance asks why last month's model bill doubled. Neither question can be answered.
Why the simple system fails
Classic logs show `POST /chat 200 14.2s`. They don't show which model was called, what context it saw, which tool returned what, how many tokens each step used, or whether the answer was any good.
Engineering concept
Record each agent run as a trace: a tree of spans for model calls, retrievals and tool calls, each with inputs, outputs, tokens, cost, latency and errors, plus quality scores attached afterwards.
Technology
OpenTelemetry spans with GenAI semantic conventions, a collector with redaction, a trace store or LLM observability platform, online evals and user feedback linked by trace id.
Trade-offs
Full prompt logging makes debugging easy and turns your trace store into a sensitive database. Sampling, redaction, retention limits and access control trade debuggability for cost and privacy.

The problem

2 min

Monday, 9:05. The booking agent handles about 200,000 runs a day.

Maya · Product manager
A guest: "The agent took 14 seconds and booked 15 November. I said the 5th."
Karim · CTO
And the model bill went from 19,000 € in August to 38,000 € in September. Why?
Lina · Senior engineer, your mentor
Your call. What can our logs tell you?

The big idea

4 min

From a log line to a run you can read

The big idea

Trace every step of a run: a wrong answer usually starts several steps before the final message.

"Landed 14 minutes late" explains nothing; a flight recorder does. A trace is the flight recorder of one run: which model, which context, which tool, which arguments, in which order.

How it works

5 min

A trace is one run; a is one step inside it (a model call, a retrieval, a tool call) with timing, inputs, outputs and status. The runtime already has the hooks: the gateway sees every model call, the tool registry every tool call.

From a run to an answer

1
Spans in the runtime
Each step opens a span with , tagged with run ID, step and the booking ID.
2
Tag what changes
Model, prompt version (), tokens and computed cost on every model span, so you can group runs by them.
3
Redact in the collector
Names, emails and card numbers removed before anything is stored.
4
Keep the right runs
Metadata on 100% of runs. Content via tail sampling (): errors, slow runs, low scores, complaints, plus 5% random.
5
Score and group
An online eval attaches a quality score to real runs (). Group cost and quality by version or model.

Watch a decision being made

4 min

Worked example

You answer Monday's two questions (Lina checks)

Explain the 14-second run that booked the 15th, and why the bill doubled, using the new traces.

1Find the run and the first wrong input
Search the booking ID, land on run_8f2c, read tool arguments in order.
text
```
├─ chat medium   step 3  1.3 s  in 9.1k out 120
│    → tool_call get_availability(date="2026-11-15")   ← guest said "the fifth"
```
Why: Lina: "Good. The booking tool did exactly what it was told." Fix the first wrong span, not the last visible one.
2Find the longest bar
create_booking took 6.1 s; its child span shows a 5.4 s database lock inside the booking API.
Why: The guest blamed the AI; the slow part was a lock. Without child spans you'd tune the wrong thing.
3Explain the bill
Group cost per 1,000 runs by prompt version: v13 3.2 €, v14 6.4 €. v14 inlined the whole FAQ: 4k → 9k input tokens per call.
Why: Cost is explainable only when spans carry tokens, cost and version.
4Fix, then keep the case
Read dates back before booking; move the FAQ back to retrieval; ticket the lock; add run_8f2c (redacted) to the eval set.
Why: Traces explain the past; evals protect the future.

In real life

3 min

Two well-known approaches to tracing

Google's Dapper: sample a little, everywhere

Its 2010 paper describes tracing with very low sampling rates
Tiny overhead; rare individual requests may be missing

Fits when patterns matter more than individual requests.

Honeycomb: keep the interesting ones

Publicly advocates wide, detailed events per request
Recommends keeping errors and slow requests, dropping boring ones

Fits when you must explain individual failures.

What goes wrong

3 min

Logging only the final answer

Errors start earlier: a bad retrieval, a wrong argument.

Instead

Every model, retrieval and tool call as spans in one tree.

Raw prompts stored forever, readable by all

The trace store becomes your least-protected copy of customer data ( applies).

Instead

Redact in the collector; short content retention; audited access.

Cost from the monthly invoice

Nobody can say which feature or prompt drives spend.

Instead

Tokens and cost per span; alert on cost per run.

Trade-offs

3 min

Option	Buys	Costs
Full content, every run	Any run explainable	Storage, privacy risk
Metadata always, content sampled	Exact metrics; content where it matters	Some runs not replayable
Metadata only	Cheap, low risk	Can't debug model behaviour

Wednesday, 14:00. Agreeing the trace retention policy.

Omar · On-call SRE
Full prompts for every run, 90 days. Complaints arrive weeks later.
Karim · CTO
200k runs a day of guests' data for three months? Metadata only.
Omar · On-call SRE
Then I can't explain a single wrong answer.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What every span carries: model, prompt version, tokens, cost, latency, status, redacted tool arguments.
What is kept and how long: metadata on 100% of runs; content tail-sampled for 14 days.
What is redacted before storage, in the collector, and who may open content.

Tell Claude Code

Instrument the booking agent with OpenTelemetry; we couldn't explain run_8f2c or a bill jump from 19,000 to 38,000 EUR. Plan first. A root span per run, a child per model, retrieval and tool call, with model, prompt_version, tokens, cost, latency, status, tool arguments; propagate context into the booking API. The collector redacts emails, phones, cards, passports before export; metadata for every run 13 months; content for errors, slow, low-score, flagged plus 5% random, 14 days, decided after the run. Show one trace and cost by prompt version.

Check before you accept

In one trace you can find the slowest span and the first wrong value.
Cost per run by prompt version shows the v13 → v14 jump in one chart.
Searching for a test guest's email finds only tokens like <EMAIL_1>.

Red flags

Head sampling at 1% "to save storage".
Unredacted prompts with no retention limit "because traces are internal".
Redaction as a nightly cleanup job after storage.

Check yourself

2 min

Q1/3Debugging

A week after run_8f2c, another guest says “I asked for the 5th and got the 15th”. This time, in the trace, the user message says “the fifth”, the model's tool call is get_availability(date="2026-11-05"), and the tool span shows the booking API received 2026-11-15. Where is the bug?

How confident are you?

Next episode: now that every span carries a cost, Karim asks what one booking conversation actually costs us, and why.

Finished?

Mark it complete to update your skill map.

Next: The LLM cost model: from tokens to a monthly bill

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Reading one run in ten seconds

Worked example

You answer Monday's two questions (Lina checks)

Explain the guest's 14-second run that booked the 15th instead of the 5th, and explain why the model bill doubled from 19,000 € to 38,000 €, using the new traces.

1Find the run
Look up the booking ID in the trace store; the root span agent.run carries it, so you land on run_8f2c.
Why: Support tickets mention bookings, not trace IDs. Putting business IDs on the root span is what makes this a one-step search.
2Find the first wrong input
Read tool arguments in order. At step 3 the model called get_availability(date="2026-11-15") after the guest said "the fifth". Everything after step 3 was correct given that input.
text
```
├─ chat medium   step 3  1.3 s  in 9.1k out 120
│    → tool_call get_availability(date="2026-11-15")   ← guest said "the fifth"
```
Why: Wrong answers usually start earlier than the final message. Lina: "Good. The booking tool did exactly what it was told."
3Find the longest bar
create_booking took 6.1 s: a timeout at 5 s, then a retry with the same key. Its child span shows a 5.4 s database lock wait inside the booking API.
Why: The guest blamed the AI; the slowest part was a database lock. Without child spans you'd tune the wrong thing.
4Explain the bill
Group cost per 1,000 runs by app.prompt.version: v13 3.2 €, v14 6.4 €. v14 inlined the whole FAQ: input per call 4k → 9k tokens.
Why: Cost is only explainable when every span carries tokens, computed cost and the version that caused them.
5Fix and keep the case
Fix: read the date back to the guest before booking, validate it against what they said; move the FAQ back to retrieval; open a ticket on the lock. Add run_8f2c (redacted) to the eval set.
Why: Traces explain the past; evals protect the future. A real failure is the best test case you'll ever get.

What to record	Where	Why	Privacy note
Model, prompt version, parameters	Every model span	Compare versions; reproduce behaviour	Safe
Input/output tokens, cost	Every model span	Cost per run, per feature, per version	Safe
Latency, TTFT, status, errors, retries	Every span	Find slow and failing steps	Safe
Tool name, arguments, result summary	Every tool span	Debug wrong actions; audit side effects	Arguments often contain PII: redact
Full prompts and outputs	Sampled model spans	Explain why the model did something	Most sensitive: redact, short retention, restricted access
Eval scores, user feedback	Attached to the run later	Quality trends; find bad runs fast	Feedback text may contain PII

Record metadata always and content selectively. The split is what makes observability affordable and lawful.

Interactive simulation

Visual diagram

Agent runtime

AI gateway

Tool services

Collector + redaction

Trace store

Online evals

User feedback

Dashboards + alerts

Click a component

From agent run to an explainable, scored, privacy-safe trace.

Analogy

An aircraft's flight recorderAn agent trace

An aircraft's flight recorderEvery control input from the pilotsAn agent traceEvery model input and output
An aircraft's flight recorderEvery instrument readingAn agent traceTool calls and their results
An aircraft's flight recorderTimestamps on every eventAn agent traceTime and cost per step
An aircraft's flight recorderInvestigators rebuilding what happenedAn agent traceDebugging a strange run
An aircraft's flight recorderLocked away, only for investigatorsAn agent traceRedacted, short-lived, restricted logs

Where the analogy breaks: a flight recorder is read after a rare crash. Agent traces are read every day, by many people, and they're full of guests' personal details. That's why you hide sensitive data before saving it.

Check your understanding

Q1/3Reasoning

A run has three model calls with 4,000, 5,200 and 6,100 input tokens and 150, 120 and 300 output tokens. Illustrative prices: €3 per million input tokens, €15 per million output tokens. What does the run cost, in euros?

€

How confident are you?

Practice ladder

Step 1 · Trace

Read a slow, wrong run

Debugging Warm-up

A guest asked for "a room for 2 adults and 1 child in Porto this weekend" and got a room for 2 people. Here's the trace:

agent.run run_41aa prompt=v15 model=medium  total 9.8 s  score: task_success=0
├─ chat medium          step 1  1.0 s  → search_hotels(city="Porto", guests=2)
├─ retrieval policies   step 1  0.3 s  k=5
├─ tool search_hotels   step 2  4.9 s  ok  12 results   (cache miss)
├─ chat medium          step 3  1.2 s  → get_availability(hotel=88, guests=2)
├─ tool get_availability step 4 0.5 s  ok
├─ chat medium          step 5  1.1 s  → ask confirmation
└─ tool create_booking  step 6  0.8 s  ok

Where did the run first go wrong?
What made it slow?
What would you fix, and what test would you add?

Step 2 · Complete

Finish the model span

Code Practice

Lina's callModel wrapper is missing the attributes that would have explained the doubled bill and let us compare prompt versions. Fill in the gaps.

return tracer.startActiveSpan("chat " + run.model, async (span) => {
  span.setAttributes({
    "gen_ai.request.model": run.model,
    "agent.run.id": run.id,
    "agent.step": step,
    ___                                          // (1)
  });
  const res = await gateway.chat({ model: run.model, messages });
  span.setAttributes({
    ___                                          // (2) usage
    ___                                          // (3) money
  });
  span.end();
  return res;
});

Step 3 · Solve

Latency crept up 40% this week

Debugging Practice

Omar: "Median booking-agent run went from 7 s to 9.8 s since Tuesday. Nothing was deployed on Tuesday… except a model upgrade at the provider and a new retrieval index."

You have traces with model, prompt version, retrieval index version and per-span latency for 100% of runs (metadata). Describe how you'd find the cause, step by step, and what you'd do with the answer.

Step 4 · Decide

Send trace content to a third-party vendor?

Decision Stretch

An LLM observability vendor offers a polished UI for traces, evals and prompt comparison. Their hosting is in the US; StayHub's guests are mostly in the EU. Today StayHub runs its own trace store in an EU region (metadata 13 months; redacted content for flagged runs, 14 days).

Options: - A: send everything (metadata + content) to the vendor. - B: send metadata only to the vendor; keep content in our EU store, linked by trace ID. - C: keep everything in-house.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a new team member what an agent trace contains, how you'd use it to debug a wrong booking, and how you keep it from becoming a privacy problem.

0 words