Skip to content
Episode 111 · The spa that closed

Citations, grounding and knowing when to say “I don't know”

Monday, 8:15: a guest asks whether Hotel Lumière's spa in Paris is open on Mondays. The concierge says "yes, 9:00–21:00" with a citation. The spa closed for renovation two weeks ago.

Giving the model the right documents doesn't mean it will use them. Here's how to get answers that cite real sources, and say "I don't know" when they can't.

30 min advancedGroundedness & citationsCitations tied to retrieved sourcesRefusing when the sources don't support an answerMeasuring groundedness and citation accuracySource freshness and stale knowledgeHallucination
  1. Problem

    A guest asks if the Paris hotel's spa is open on Mondays. The bot says “Yes, 9:00–21:00” with a citation. The spa closed for renovation two weeks ago.

  2. Why the simple system fails

    A plain RAG prompt lets the model blend retrieved text with its own guesses, cite sources that don't say what it claims, answer when nothing relevant was retrieved, and quote documents that were true last month.

  3. Engineering concept

    Treat grounding as a contract: every claim must be supported by a provided source, citations must point to the passage that supports them, missing evidence must produce a refusal or escalation, and sources must carry validity dates.

  4. Technology

    Source-tagged prompts, structured output with citation ids and quotes, code-side citation verification, freshness metadata and re-indexing, and groundedness evals (faithfulness, citation precision, refusal correctness).

  5. Trade-offs

    Strict grounding makes the assistant refuse more often and feel less helpful; verification adds latency and tokens; freshness pipelines add moving parts. You tune where on the helpful-vs-safe curve you sit, per use case.

01

The problem

2 min
Monday, 8:15. The guest concierge has answered from RAG for three weeks.
  1. A guest · StayHub customer
    Is the spa at Hotel Lumière open on Mondays?
  2. Omar · On-call SRE
    Concierge: "Yes, daily 9:00–21:00 [S2]". S2 is real. The spa closed for renovation on 22 September.
  3. Maya · Product manager
    And this week's audit: in 11% of cited answers, the source doesn't say what the answer claims.
  4. Lina · Senior engineer, your mentor
    You lead this one.
02

The big idea

4 min
A citation isn't a proof
The big idea

A citation shows where a claim came from. Checks in code, an honest "I can't find that" and fresh sources show it's true.

Two different things broke. The model went beyond its sources (the 11%), and a source was stale (the spa). A better prompt fixes neither.

03

How it works

5 min
Four parts of a trustworthy answer, and how to measure them
  1. 1
    Grounding

    A grounded answer has every factual claim supported by a provided source, not by the model's training or guesses (). Sources are tagged with ids and dates.

  2. 2
    Checked citations

    Each claim carries an and a short verbatim quote. Code checks the id exists, the quote is in that source, and the claim's numbers are in the quote. Failing claims are dropped, never shown.

  3. 3
    Refusal as a handoff

    Abstention (): when sources don't answer, say "I can't find that" and hand over to the front desk with the question attached.

  4. 4
    Freshness

    Freshness: CMS change events re-index within 15 minutes, replaced chunks are deleted, and chunks carry valid-from / valid-to dates so expired ones are filtered out.

  5. 5
    Measure both sides

    Track and unsupported claims, and refusal correctness on unanswerable questions. Grounding moves you along a curve between helpful and safe.

04

Watch a decision being made

4 min
Worked example

You make the concierge's answers checkable; Lina reviews

Stop stale and unsupported answers like the spa: every claim backed by a current source, checked in code, with a clean handoff otherwise.

  1. 1Ask for claims, not prose

    Structured output: status, a list of claims (text, source ids, quote), or insufficient.

    json
    { "status": "answered",
  "claims": [{ "text": "The spa is closed for renovation until 31 October.",
    "source_ids": ["S1"], "quote": "spa closed for renovation 22 Sept – 31 Oct" }] }

    Why: A paragraph can't be checked claim by claim; a list of claims with quotes can.

  2. 2Verify in code

    Ids exist, quotes appear in the cited source, numbers appear in the quote. Otherwise the claim is dropped.

    Why: Most of the 11% were fees and times drifting from the source. A substring check catches them for free.

  3. 3Make refusal a handoff

    "I can't confirm that from Hotel Lumière's information. I've asked the front desk, who'll reply here."

    Why: A refusal that leads somewhere is a feature; a guess is a promise we can't keep.

  4. 4Keep sources fresh, measure both sides

    Re-index on CMS events within 15 minutes, filter expired chunks. Unsupported claims 11% → 2%; refusals 3% → 9%. Lina: "Checked. Ship it."

    Why: Citation checks can't catch a source that was true in August; only freshness can.

The whole thing in one line: Trust came from checkable claims, a useful refusal and fresh sources, and the price, more refusals, was measured and accepted on purpose.
05

In real life

3 min
Showing sources vs checking them

Google AI Overviews at launch (2024)

  • AI summaries above search results, with links to sources
  • Widely reported early answers drew on joke posts, such as glue in pizza sauce
  • A linked source wasn't necessarily reliable or serious

Linking a source isn't the same as trusting it.

Wikipedia: verifiability

  • Core policy: claims must be attributable to reliable published sources
  • "Citation needed" flags; unsupported claims get removed
  • Slower and stricter, and trusted for it

Checked sources, and removing what can't be supported.

06

What goes wrong

3 min
Treating a citation as proof

It shows a source was in the context, not that it says what the claim says.

Instead

Require quotes; check ids, quotes and numbers in code.

Banning or omitting "I don't know"

Gaps become invented promises.

Instead

An explicit "insufficient" status that becomes a handoff; use the refusal log to fill missing documents.

Ignoring freshness, or evaluating only answerable questions

A correct citation to last month's page is still wrong; a model that always answers looks perfect.

Instead

Re-index on change with validity dates; add unanswerable questions to the eval.

07

Trade-offs

3 min
GroundingChoose whenYou pay
StrictAnswers are promises: prices, policies, hours, safetyMore refusals; handoff and freshness work
LooseLow-stakes, clearly labelled suggestionsInvented facts; citations borrow trust
Strict + verifier + handoffCustomer-facing answers at scaleMost moving parts; evals to maintain
Monday, 11:00. After reverting the prompt line.
  1. Maya · Product manager
    9% "I can't find that" feels broken. Allow claims without quotes when the model is confident.
  2. Omar · On-call SRE
    "Confident" is how we got the spa. Each wrong promise is a refund and a bad review.
  3. Maya · Product manager
    And each refusal is a guest waiting for the front desk.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • What the concierge does when no source answers: "I can't find that" plus the front desk. You accept refusals rising (3% → 9%) to cut unsupported claims (11% → 2%).
  • Which claims need a verbatim quote checked in code before display: prices, dates, opening hours, fees.
  • How fresh sources must be: re-indexed within 15 minutes, old versions deleted, expired chunks filtered.
Tell Claude Code
The concierge told guests Hotel Lumière's spa was open while it was closed for renovation: an old page was still indexed, the notice never was. Make the answer step return claims, each with source ids and a verbatim quote, or `status: insufficient`. Add a verifier in code that drops claims whose quote isn't in the cited chunk and shows the handoff when nothing is left. Re-index on CMS change events within 15 minutes; filter on `valid_to`. Plan first, then rerun the audit with 100 unanswerable questions.
Check before you accept
  • Ask about the Lumière spa in October: it says closed and cites the renovation notice, with its date.
  • The verifier checks the quote really appears in the cited chunk, not just that the id exists.
  • Audit numbers read as a set: unsupported claims (~2%), correct refusals, over-refusals.
Red flags
  • "Never hallucinate" added to the prompt and called solved, with no check in code.
  • An answer counted as grounded because it has a citation.
  • Refusal rate tuned down without showing what happened to unsupported claims.
09

Check yourself

2 min
Q1/3Debugging
An answer says “Parking costs €35 per night [S3]”. S3 was in the context and is about parking, but says “€35 per day, max €70 per stay”. What failed, and what catches it?
How confident are you?

Next episode: a guest types "find and book me a quiet hotel in Lisbon next weekend under €200", and the concierge can only describe hotels. It can't check a room or book anything.

Finished?

Mark it complete to update your skill map.

Next: LLM vs AI agent vs agentic system
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Claims, quotes, checks
Worked example

You make the concierge's answers checkable

Stop stale and unsupported answers like the spa: every claim supported by a current source, checked in code, and a clean handoff when sources don't cover the question. Lina reviews your result.

  1. 1Tag and date every source

    Each retrieved chunk becomes <source id="S1" title="…" updated="2026-09-22" valid_to="2026-10-31">. The prompt says: use only these, cite each claim with a verbatim quote of up to 25 words, return insufficient if they don't answer.

    Why: Ids make claims checkable; dates let the model, the guest and our filters see how current a source is.

  2. 2Ask for claims, not prose

    Structured output: status, a list of claims (text, source_ids, quote), and missing when insufficient.

    Why: A paragraph can't be checked claim by claim; a list of claims with quotes can.

  3. 3Verify in code

    Each cited id must exist; each quote must appear in that source; numbers in the claim must appear in the quote.

    ts
    const ok = claim.source_ids.every((id) => byId.has(id)) &&
  claim.source_ids.some((id) => byId.get(id)!.includes(norm(claim.quote))) &&
  numbersIn(claim.text).every((n) => claim.quote.includes(n));
if (!ok) dropped.push(claim);   // never shown

    Why: Most of the 11% were numbers (fees, times) drifting from the source. A substring check catches them for free.

  4. 4Make refusal a handoff

    If status is insufficient, or every claim was dropped: "I can't confirm that from Hotel Lumière's information. I've asked the front desk, who'll reply here." The ticket carries the question and what was missing.

    Why: A refusal that leads somewhere is a feature; a guess is a promise we can't keep.

  5. 5Keep sources fresh

    CMS change events re-index within 15 minutes; replaced chunks are deleted; news posts about closures are indexed with valid_from/valid_to; retrieval filters out expired chunks.

    Why: Citation checks can't catch a source that was true in August. Only freshness can.

  6. 6Measure both sides

    Nightly eval: 500 answerable + 150 unanswerable questions. Unsupported claims 11% → 2%; refusals 3% → 9%, with refusal correctness tracked. Lina: "Checked. Ship it."

    Why: Grounding moves you along a curve between helpful and safe; you need both numbers to know where you are.

The whole thing in one line: You tagged and dated sources, asked for claims with quotes, verified them in code, turned refusals into front-desk handoffs, kept the index fresh with validity dates, and measured both unsupported claims (11% → 2%) and refusals (3% → 9%).
MetricQuestion it answersHow to measure
FaithfulnessIs every claim supported by the cited sources?Code checks (quotes, numbers) + an LLM judge on a sample, checked against people
Citation precisionDo cited sources actually support their claims?Labelled audit of cited claims
Refusal correctnessDoes it refuse exactly when sources don't answer?Answerable + unanswerable eval questions
FreshnessHow long until a CMS change is searchable?Time from change event to indexed chunk (target ≤ 15 min)
Groundedness is engineered and measured, never assumed from the presence of citations.
11

Interactive simulation

12

Visual diagram

Click a component

A grounded answering pipeline with verification and a freshness loop. Click components for details.

13

Analogy

  • A junior lawyer writing a memoA binder of case filesA grounded AI answerThe retrieved sources
  • A junior lawyer writing a memoA footnote on every statementA grounded AI answerInline citations
  • A junior lawyer writing a memo"The files don't cover this"A grounded AI answerRefusing when evidence is missing
  • A junior lawyer writing a memoChecking the date on each fileA grounded AI answerKeeping sources fresh
  • A junior lawyer writing a memoA partner spot-checking footnotesA grounded AI answerCitation checks and evals

Where the analogy breaks: a junior lawyer feels embarrassed when caught guessing. A model doesn't, so refusing must be designed in and tested, not hoped for.

14

Why it matters

RAG usually goes exactly where wrong answers are expensive: policies, prices, legal terms, safety rules.

A confident wrong answer costs more than no answer. Tell a guest "free cancellation until 24 hours before" when the rate was non-refundable, and they'll demand a refund. They might even be legally right.

Citations also change behaviour. Once people see sources, they trust more and check less. Bad citations exploit that.

And stale sources are invisible. The answer is perfectly grounded in a document that's no longer true. Nothing in the reply gives it away.

The goal isn't "never wrong". It's: right when it answers, honest when it can't, and you can prove both.

15

Check your understanding

Q1/3Design
Your eval set has 400 answerable questions and scores 96% correctness. Why is this not enough to ship a policy assistant?
How confident are you?
16

Practice ladder

Step 1 · Trace

Verify three claims

Debugging Warm-up

Sources given to the model:

  • S1 (updated 2026-09-22, valid to 2026-10-31): "Spa closed for renovation 22 Sept – 31 Oct. The pool remains open 7:00–22:00."
  • S2 (updated 2026-08-04): "Late checkout until 14:00 costs €30, subject to occupancy."

The model's claims: 1. "The pool is open 7:00–22:00." — S1, quote: "The pool remains open 7:00–22:00" 2. "Late checkout until 14:00 costs €40." — S2, quote: "Late checkout until 14:00 costs €30" 3. "The spa reopens with a new sauna." — S1, quote: "Spa closed for renovation 22 Sept – 31 Oct"

Trace the verifier (ids exist, quote appears in the source, the claim's numbers appear in the quote). Which claims are shown?

Step 2 · Complete

Finish the verifier

Code Practice

Complete the two missing checks and the final decision in Lina's verifier.

type Claim = { text: string; source_ids: string[]; quote: string };
type Answer = { status: "answered" | "insufficient"; claims: Claim[] };

export function verify(answer: Answer, sources: { id: string; text: string; validTo?: string }[], today: string) {
  const byId = new Map(sources.map((s) => [s.id, s]));
  const kept = answer.claims.filter((c) => {
    const cited = c.source_ids.map((id) => byId.get(id)).filter(Boolean);
    if (cited.length !== c.source_ids.length) return false;          // unknown id
    // TODO 1: the quote must appear in at least one cited source
    // TODO 2: no cited source may be expired (validTo before today)
    return numbersIn(c.text).every((n) => c.quote.includes(n));
  });
  // TODO 3: decide what the guest sees
}
Step 3 · Solve

Keep 400 policy changes a week fresh

Design Practice

Hotels change policies about 400 times a week in the CMS: prices, opening hours, closures, sometimes with a start and end date ("pool closed 3–9 Nov"). Some changes are posted as news items, not on policy pages. Guests must never get an answer based on a replaced or expired policy.

Design the freshness pipeline: triggers, what happens to old chunks, how dates are handled, and how you'd know it works.

Step 4 · Decide

How strict for restaurant tips?

Decision Stretch

Maya wants the concierge to suggest restaurants near each hotel. StayHub's sources cover only about 30% of neighbourhoods (curated lists by hotels). The model knows many well-known restaurants from training, but can't know closures or current hours.

Options: A. Strict: only curated sources, otherwise "I don't have recommendations here; the front desk can help". B. Loose for this use case: allow the model's own suggestions, clearly labelled as general tips, never with hours, prices or bookings.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

17

Explain it yourself

A product manager says “our bot shows citations, so it's grounded.” Explain what's missing and what you'd build and measure.

0 words