Episode 103 · The booking reference nobody wrote

Structured output: making LLMs return data your code can trust

Wednesday, 7:40: last night 750 of 25,000 guest emails crashed the import because the model's JSON didn't parse. Worse, one email moved the check-out date on someone else's booking, using a booking reference the guest never wrote.

Models write text. Your code needs data. Here's how to get answers back in a fixed shape you can actually trust.

30 min intermediateStructured output from LLMsConstrained decoding (schema-enforced generation)Validate, repair and retry model outputDesigning schemas the model can fill wellTool schemas & descriptionsValidating model output before acting

Problem
Your code needs fields (intent, booking reference, dates), but an LLM produces prose.
Why the simple system fails
Asking nicely for JSON works 97% of the time. The other 3% (code fences, extra commentary, renamed fields, truncated output, invented values) crashes the pipeline or silently writes garbage.
Engineering concept
Treat model output as untrusted input: define a schema, make generation follow it where possible, then validate structure and meaning, retry with the error, and fall back to a human.
Technology
JSON Schema, provider structured-output modes (constrained decoding), forced tool calls, Zod or Pydantic validation, retry loops.
Trade-offs
Stronger guarantees cost schema restrictions, some latency on first use, and can hurt reasoning if the schema forces an answer too early. Syntax guarantees never prove the values are right.

The problem

2 min

Wednesday, 7:40. The overnight import of guest emails into the booking system.

Omar · On-call SRE
750 of 25,000 emails failed to parse: code fences, a "Sure! Here's the JSON:" line, a field called bookingRef.
Omar · On-call SRE
Worse: a guest asked to move "my Lisbon stay", with no reference. The ticket said LIS-2208, someone else's booking, and we moved it.
You · Engineer at StayHub
So we switch on strict JSON mode and both go away?

The big idea

4 min

Shape is not truth

The big idea

The schema guarantees the shape; only your code checks the truth.

A form at the front desk has one box for the booking number, so nobody writes a poem in the date box. But a guest can still write the wrong number in the right box. The receptionist checks it against the reservation book.

How it works

5 min

Four levels of structured output, weakest first

1
Prompt-only
"Reply with JSON like {…}". Usually works; fails as code fences, chatty lines, renamed fields: our 3%.
2
JSON mode
The output always parses as . Your fields may still be missing or renamed.
3
Schema-constrained
You send a (which fields, types, allowed values). lets the model pick only tokens that keep the output valid. Unless it's cut off at max_tokens.
4
Your validation
Semantic validation in your code checks meaning: the reference appears in the email and exists in our system, check-out is after check-in and in the future.
5
Required means "invent if needed"
Our schema made booking_ref required. With no reference in the email, the model had to fill the box, and wrote a plausible one. A nullable field lets it say "not stated".

Watch a decision being made

4 min

Worked example

You rebuild the triage extractor

No crashes, no invented booking references, and a clear path for anything that fails. Lina asks one question per step.

1Design the schema for honesty

Lina: "What should it write when the email has no reference?" You: nullable, an other intent, evidence first.

export const Ticket = z.object({
  evidence: z.string().describe("Short quotes from the email"),
  intent: z.enum(["cancel", "change_dates", "invoice", "complaint", "other"]),
  booking_ref: z.string().nullable()
    .describe("Exactly as written in the email. null if absent. Never guess."),
  new_check_out: z.string().date().nullable(),
});

Why: Without an "unknown" option, the model must invent. And it writes left to right: quoting the email first puts the facts in front of the decisions.

2Enforce the shape
The provider's strict mode, plus two checks the schema can't do: cut off at max_tokens, or a refusal.
Why: This removes the 750 parse failures. Truncation voids the guarantee.
3Check the meaning
booking_ref must appear in the email and exist in our system; dates in order, not in the past.
Why: Only our code knows what's true. This is what stops LIS-2208.
4Retry once, then a person
Retry once with the exact error ("LIS-2208 does not appear in the email; use null"), then the hotel's staff queue.
Why: A specific error gives the retry something new. A blind second retry rarely helps and doubles cost.

In real life

3 min

Enforce while generating, or validate after?

OpenAI's Structured Outputs

Launched in 2024: send a JSON Schema in strict mode and the output is constrained to match
No parse failures; a subset of JSON Schema; only that provider's models

Fits when your provider supports it and the shape must never break.

Instructor (open-source): validate, then retry

A popular library that checks output against a Pydantic model and retries with the errors
Works with almost any model; a small failure rate remains

Fits when you need portability across models.

What goes wrong

3 min

Trusting values because the shape is valid

A perfect object can hold a reference the guest never wrote.

Instead

Validate meaning against the email and your system of record.

Decision fields before the evidence

It's decided before any reasoning tokens exist.

Instead

Evidence first, then classification, then derived values.

Changing the schema without re-testing

A renamed field or new enum value shifts the other fields too.

Instead

Version schemas; run the per field before shipping.

Trade-offs

3 min

Approach	Guarantees	Fits when
Prompt-only JSON + repair	Nothing by itself	Models without structured-output support
Schema-constrained	Shape: fields, types, enums	Default for extraction and classification
Forced	Arguments match the tool schema	The app already runs a tool-calling loop

Monday, 14:00. Planning next quarter.

Omar · On-call SRE
Keep the provider's strict schema mode. I never want a 3 a.m. import crash again.
Karim · CTO
Next quarter we may move triage to a cheaper self-hosted model with no strict mode.
Karim · CTO
Build validate-and-repair now and stay portable.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The ticket schema: which fields exist, the allowed intents, and which fields may be null (a missing booking ref is null, never a guess).
Which meaning checks run before anything is written: the ref appears in the email and exists in the booking system; dates in order.
Retries: one, with the specific error, then the staff queue.

Tell Claude Code

Our email triage asks for JSON in the prompt; 3% of 25,000 daily emails fail to parse, and ~100 a day get a booking_ref the email doesn't contain (how LIS-2208 got the wrong check-out). Switch to schema-constrained output with a Zod schema: `intent` enum with `other`, `booking_ref` and dates nullable. Check in code: the ref appears verbatim in the email and exists in the PMS; dates valid. On failure, retry once with the error, then the staff queue. Plan first, then replay yesterday's emails and give me counts.

Check before you accept

booking_ref is nullable, described "null if absent, never guess".
An email with no reference yields booking_ref: null.
The replay report shows parse failures (target 0), refs caught, and ~0.5% to staff.

Red flags

It says "the schema guarantees correct data now" and drops the meaning checks.
It adds a regex and calls invented refs solved: made-up refs match too.
It retries blindly with no limit.

Check yourself

2 min

Q1/3Reasoning

You switched to schema-constrained output. Parse errors went to zero. Which bug can still happen?

How confident are you?

Next episode: on turn 1 the concierge is flawless; on turn 60 it changes the wrong one of a guest's two bookings and forgets a price it promised on turn 12.

Finished?

Mark it complete to update your skill map.

Next: Context engineering: deciding what the model gets to see

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Schema, validation, one retry

Worked example

You rebuild the triage extractor

Turn guest emails into tickets the import can trust: no crashes, no invented booking references, and a clear path for anything that fails. Lina asks one question per step; you answer it.

1Design the schema for honesty

Lina: "What should the model write when the email has no reference?" You: make it nullable, add other to the intent enum, and describe each field.

export const Ticket = z.object({
  evidence: z.string().describe("Short quotes from the email"),
  intent: z.enum(["cancel", "change_dates", "invoice", "complaint", "other"]),
  booking_ref: z.string().regex(/^[A-Z]{3}-\d{4}$/).nullable()
    .describe("Exactly as written in the email. null if absent. Never guess."),
  new_check_in: z.string().date().nullable(),
  new_check_out: z.string().date().nullable(),
  urgency: z.enum(["low", "normal", "high"]),
});

Why: Without an "unknown" option, the model must invent something to satisfy the shape.

2Order fields: evidence first
Lina: "Which field does the model write first?" You: evidence, a quote from the email, then the decisions.
Why: Generation goes left to right. If intent comes first, it's decided before any reasoning tokens exist.
3Enforce the shape
Send the schema with the provider's strict structured-output mode, and check two things the schema can't: the output was cut off at max_tokens, or the model refused.
Why: Constrained decoding removes the 750 parse failures. A truncated output voids the shape guarantee.
4Check the meaning
booking_ref must appear in the email and exist in our system; new_check_out must be after new_check_in and not in the past; change_dates needs a date.
Why: Only our code knows what's true. This check is what stops LIS-2208.
5Retry once, then a person
On failure, retry once with the exact errors ("booking_ref LIS-2208 does not appear in the email; use null"). If it fails again, send it to the hotel's staff queue with the draft filled in.
Why: A specific error gives the retry something new to work with. A second blind retry rarely helps and doubles cost.

Approach	Guarantees	Does not guarantee	Typical use
Prompt-only JSON	Nothing	Parsing, fields, types	Prototypes, models without other options
JSON mode	Valid JSON syntax	Your fields, enums, types	Loose payloads, older APIs
Schema-constrained	Shape: required fields, types, enums	Correct values, no truncation	Default for extraction and classification
Forced tool call	Arguments match the tool schema (when strict)	Correct values	When you already use
Any of the above + validation	Shape and the rules you encode	Facts you didn't write a rule for	Production

Interactive simulation

Visual diagram

Inbound email

Extraction worker

LLM (schema-constrained)

Validate

Tickets DB

Human review queue

PMS lookup

Click a component

From guest email to a trusted ticket: generate against a schema, validate shape and meaning, retry once with the error, then fall back to a human.

Analogy

At the hotel front deskStructured output

At the hotel front desk"Write down what you need" gets a letterStructured outputFree-text model output
At the hotel front deskA form with labelled boxesStructured outputA JSON Schema
At the hotel front deskTick-boxes for the request typeStructured outputA fixed list of allowed values
At the hotel front deskA pen that can't write outside the boxesStructured outputConstrained decoding
At the hotel front deskThe clerk checking the formStructured outputYour validation code

Where the analogy breaks: a guest who doesn't know their booking number leaves the box empty. A model may fill it with something believable, so give it a way to say "not found".

Check your understanding

Q1/4Design

Order the steps of a robust extraction pipeline:

1.On failure, retry once with the specific errors
2.Parse and validate against the schema (Zod/Pydantic)
3.Call the model with the JSON Schema (structured-output mode)
4.Still failing: route to a human queue with the draft and errors
5.Check the stop reason (truncated? refused?)
6.Run semantic checks against the source text and systems of record

How confident are you?

Practice ladder

Step 1 · Trace

Trace three outputs through the pipeline

Debugging Warm-up

The email: "Hi, I'm booked under LIS-2280, arriving 12 July. Could I leave on the 14th instead of the 16th? Thanks, Ana". Today is 1 July.

The pipeline: parse → schema check → semantic checks (ref in email, check-out after check-in, dates not in the past) → retry once with errors → staff queue.

Three possible model outputs: - A: Sure! {"intent":"change_dates", ...} (prompt-only mode) - B: {"evidence":"leave on the 14th","intent":"change_dates","booking_ref":"LIS-2208","new_check_in":null,"new_check_out":"2026-07-14","urgency":"normal"} - C: {"evidence":"leave on the 14th instead of the 16th","intent":"change_dates","booking_ref":"LIS-2280","new_check_in":null,"new_check_out":"2026-07-14","urgency":"normal"}

Trace each one: where does it stop or pass, and what happens next?

Step 2 · Complete

Finish the semantic checks

Code Practice

Lina's semantic checks have two gaps. Write the missing rules.

function semanticErrors(t: Ticket, emailBody: string, today: string): string[] {
  const errs: string[] = [];
  if (t.booking_ref && !emailBody.includes(t.booking_ref))
    errs.push(`booking_ref ${t.booking_ref} does not appear in the email; use null if absent`);
  if (t.new_check_in && t.new_check_in < today)
    errs.push("new_check_in is in the past");
  // TODO 1: check-out must be after check-in (when both are present)
  // TODO 2: intent "change_dates" needs at least one new date
  return errs;
}

Step 3 · Solve

Turn search text into filters

Design Practice

Maya wants guests to type searches like "quiet room near the old town in Porto, under €150, 2 adults, next weekend" and get proper filters. The search API accepts: city (one of StayHub's 380 cities), max price per night, adults, children, check-in and check-out dates, and up to 5 amenity tags from a fixed list of 40.

Design the schema (with what's nullable and why), the checks your code runs, and what happens when the request is vague.

Step 4 · Decide

Strict schema for complaint severity?

Decision Stretch

Complaints must be rated severity 1–5 (5 = safety issue, sent to the hotel manager within 10 minutes). Today one strict schema call returns {"severity": 4, "category": "cleanliness"}. On the golden set of 300 complaints, severity is right 81% of the time; safety cases are missed 1 time in 12.

A teammate proposes: step 1, a free-text call where the model reasons about the complaint; step 2, a cheap strict-schema call that extracts severity and category from that reasoning. Cost goes from about €0.002 to €0.004 per complaint; 6,000 complaints a month.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a teammate why 'we use structured outputs, so the data is correct' is wrong, and what a production extraction pipeline looks like.

0 words