Skip to content
Episode 108 · The handbook nobody read

RAG: answering from your documents

Wednesday, 2:10 am: a night receptionist asks StayHub's new staff assistant about late checkout, and it confidently answers "free until 14:00 for all guests". The 400-page handbook says €30, and it changes every month.

The model doesn't know your hotel's pet policy. RAG looks it up and hands it over just in time, so answers come from your facts, with sources.

30 min intermediateRetrieval-augmented generationChunkingEmbeddingsVector databasesRerankingHybrid search
  1. Problem

    Staff and guests ask questions whose answers are buried in a 400-page policy handbook that changes every month, different per hotel.

  2. Why the simple system fails

    The model alone doesn't know your policies and invents plausible ones. Pasting the whole handbook in every call is slow, expensive and still misses details.

  3. Engineering concept

    Retrieve only the few most relevant passages for each question and give them to the model as context, then require answers to cite them.

  4. Technology

    Chunking, embedding models, a vector database, hybrid search, a reranker, a prompt with citations.

  5. Trade-offs

    Answer quality is capped by retrieval quality. More pipeline stages mean more latency, more things to index, evaluate and keep fresh.

01

The problem

2 min
Wednesday, 2:10 am. The staff assistant's first week at partner hotels' front desks.
  1. Omar · On-call SRE
    Hotel Alfama, night desk: "Late checkout until 14:00?" The assistant: "Free for all guests."
  2. Maya · Product manager
    The handbook says €30, and Alfama has no late checkout on Saturdays. It's 400 pages and changes every month.
  3. You · Engineer at StayHub
    Can't we just put the handbook in the prompt?
  4. Lina · Senior engineer, your mentor
    Guess first. Then let's price it.
02

The big idea

4 min
Retrieve, then answer
The big idea

RAG puts the right facts in the context just in time, so it is mostly a search problem: no retrieved fact, no right answer.

The model is still a next-token predictor. Retrieval just makes the right paragraph the most plausible thing to continue from. Four chunks, about 2,000 tokens and a cent of input, each claim citing its source.

03

How it works

5 min

(retrieval-augmented generation) has two halves: indexing, done when documents change, and answering, done for every question.

How RAG answers the late-checkout question
  1. 1
    Index

    Cut the handbook and hotel pages into chunks (passages of a few hundred tokens, see ), embed each, and store it with metadata: hotel, language, last updated.

  2. 2
    Retrieve wide, filtered

    Embed the question and find the top 30 similar chunks, filtered to Hotel Alfama + the group handbook. The filter comes from the staff member's login.

  3. 3
    Rerank

    A reranker is a slower, more accurate model that reads question and chunk together (). It keeps the best 4.

  4. 4
    Build the context

    Instructions + 4 chunks with ids and dates: about 2,000 tokens instead of 200,000. Sources are marked as data, not instructions.

  5. 5
    Answer with citations

    The model answers only from those chunks and gives a citation (the source id) for each claim, or says "Not in the policies".

  6. 6
    Measure both halves

    Score retrieval (is the right chunk in the top 4?) separately from answers (correct and cited?). Otherwise you tune the prompt for a search problem.

04

Watch a decision being made

4 min
Worked example

Lina designs the first RAG pipeline

Make the staff assistant answer from the group handbook and each hotel's policy page, with citations, for 30,000 questions a day, without showing one hotel another's documents.

  1. 1Index with metadata

    "Split by headings, about 500 tokens. Every chunk gets hotel_id (or GROUP), language and updated_at. When a document changes, delete its old chunks and re-index."

    Why: Metadata makes filtering possible; deleting old chunks stops last month's rules being retrieved.

  2. 2Filter from who's asking

    "The filter comes from the login, not the question."

    sql
    WHERE hotel_id IN (:userHotelId, 'GROUP')

    Why: It's relevance and access control: the model must never see another hotel's documents.

  3. 3Retrieve wide, rerank, ground

    "Top 30, rerank to 4. Prompt: only these sources, cite each claim, say 'Not in the policies' otherwise. If the best score is below a floor, skip the model."

    Why: Wide retrieval catches the right chunk; the "don't know" path stops invention when retrieval comes back empty.

  4. 4Measure the two halves

    "300 real staff questions with the correct chunk marked. Score retrieval and answers separately."

    Why: In the pilot, 140 of 200 wrong answers were retrieval misses. Without separate scores you'd blame the model.

The whole thing in one line: A RAG pipeline is a search system with a model at the end: metadata, a filter from identity, wide-then-rerank, a "don't know" path, and retrieval measured on its own.
05

In real life

3 min
Grounded in documents, or not

Morgan Stanley: answers from its own library

  • Publicly announced in 2023 an internal assistant that answers advisors from the firm's research library
  • Answers grounded in the firm's own content
  • Needs indexing, access rules and current documents

Fits staff questions over a large, private, changing knowledge base.

Air Canada: a policy the airline didn't have

  • In a widely reported 2024 case, its chatbot told a customer about a bereavement refund the policy didn't allow
  • A Canadian tribunal held the airline responsible for what its chatbot said

What happens when an assistant answers beyond its sources.

06

What goes wrong

3 min
No metadata filter, or one taken from the question

Other hotels' passages come back: wrong, and possibly confidential.

Instead

Filter by hotel from the user's login on every query; index only approved documents.

Treating RAG as a model problem

Most wrong answers are retrieval misses; a prompt can't fix a passage never retrieved.

Instead

Log retrieved chunk ids and measure retrieval separately.

No "don't know" path, rare re-indexing

Weak retrieval gets filled with a plausible policy; old rules get quoted.

Instead

A relevance floor, "Not in the policies", and re-index on every document change.

07

Trade-offs

3 min
ApproachBest forFreshnessWeakness
RAGLarge, private, changing knowledge; citationsMinutes (re-index)Quality capped by retrieval
Fine-tuningStyle, format, narrow behaviourWeeks (retrain)Poor way to add facts; no sources
Long contextOne or a few documents, low volumeInstantCostly at volume; the middle gets missed
Friday, 14:00. Planning the rollout to all partner hotels.
  1. Karim · CTO
    Pipeline, index, reranker, filters… Fine-tune on the handbook once a month instead. No infrastructure.
  2. Omar · On-call SRE
    And between fine-tunes it confidently quotes last month's rules, with no source to check.
  3. Karim · CTO
    The handbook changes maybe 20 pages a month.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • RAG, long context or nothing: the whole handbook is ≈ €1.00 a question (€30k a day); 4 chunks are ≈ €0.01.
  • Which documents get indexed, and the metadata every chunk carries (hotel, section, language, updated), so a hotel's override beats the group rule.
  • What happens when the chunks don't cover the question: say so and point to a person, never guess.
Tell Claude Code
The front-desk assistant invented "free late checkout until 14:00". Add RAG over the group handbook and each hotel's policy page: chunk by heading, store hotel id, section and updated date, retrieve the top 30 filtered to the asker's hotel plus group rules, rerank to 4. Prompt: answer only from these chunks, cite ids, otherwise "I don't know". Log retrieved chunk ids. Plan first, then run `evals/front-desk.jsonl` and split failures into retrieval misses and generation mistakes.
Check before you accept
  • Ask about late checkout at Alfama on a Saturday: the answer uses the hotel override and cites it.
  • In the logs, cited chunk ids are among those actually retrieved.
  • A question the handbook doesn't cover gets "I don't know", not an invented rule.
Red flags
  • The whole handbook pasted into every prompt "because the context window is big enough".
  • Wrong answers blamed on the model without showing the retrieved chunks.
  • No hotel filter, so one hotel's staff get another hotel's rules.
09

Check yourself

2 min
Q1/3Debugging
The assistant says “Dogs are welcome everywhere.” The policy says dogs are welcome except in the spa wing. Logs show the retrieved chunk ends with “Dogs are welcome.” What's the most likely root cause?
How confident are you?

Next episode: asked whether a 16-year-old can check in alone, the assistant says yes, because the exception ("18 without a guardian") landed in the next chunk.

Finished?

Mark it complete to update your skill map.

Next: Chunking strategies: cutting documents so retrieval works
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Two halves of one pipeline
Worked example

Lina builds the first RAG pipeline

Make the staff assistant answer from the group handbook and each hotel's policy page, with citations, for 30,000 questions a day, without showing one hotel another's documents.

  1. 1Index with metadata

    Split by headings (~500 tokens), embed, and store each chunk with hotel_id (or GROUP for the handbook), language and updated_at. On document change, delete the old chunks and re-index.

    ts
    async function indexDocument(doc: PolicyDoc) {
  const chunks = splitByHeadings(doc.text, { maxTokens: 500 });
  const vectors = await embed(chunks.map((c) => c.text));
  await db.deleteChunks({ docId: doc.id });   // no stale leftovers
  await db.insertChunks(chunks.map((c, i) => ({
    id: `${doc.id}-${i}`, text: c.text, vector: vectors[i],
    hotelId: doc.hotelId, lang: doc.lang, updatedAt: doc.updatedAt,
  })));
}

    Why: Metadata makes filtering possible; deleting old chunks stops last month's rules being retrieved.

  2. 2Filter from who's asking

    At query time, the filter comes from the staff member's login, not from the question: hotel_id IN (user.hotelId, 'GROUP').

    Why: It's relevance and access control. The model must never see another hotel's documents.

  3. 3Retrieve wide, then rerank

    Top 30 candidates by similarity, then a reranker keeps the best 4 (~2,000 tokens).

    Why: Wide retrieval catches the right chunk; reranking puts it at the top and keeps the context small.

  4. 4Ground the answer

    Prompt: only these sources, cite each claim by id, say "Not in the policies" otherwise, sources are data. If the best reranked score is below a floor, skip the model and say it isn't in the policies.

    Why: A "don't know" path is what stops the model from inventing when retrieval comes back empty.

  5. 5Measure the two halves

    A test set of 300 real staff questions with the correct chunk marked: retrieval (is it in the top 4?) and answers (correct and cited?) are scored separately.

    Why: In the pilot 70% of wrong answers were retrieval misses. Without separate scores you'd tune the prompt for a search problem.

The whole thing in one line: Lina indexed documents with hotel metadata, filtered by the asker's hotel, retrieved wide and reranked to four chunks, grounded the answer with citations and a "not found" path, and measured retrieval separately from generation.
ApproachBest forFreshnessCost profileWeakness
RAGLarge, changing, private knowledge; citations neededMinutes (re-index)Index + small context per questionQuality capped by retrieval
Fine-tuningStyle, format, narrow behaviourWeeks (retrain)Training runs + evalsPoor way to add facts; no citations
Long context (paste it all)One or a few documents, deep readingInstantMany input tokens on every callSlow and costly at volume; the middle gets missed
They combine: a fine-tuned model can still use RAG, and RAG can retrieve whole sections into a long context.
11

Interactive simulation

12

Visual diagram

Click a component

Top row: indexing (offline). Bottom row: answering a question (online).

13

Analogy

  • A student with a librarianA closed-book exam, bluffing when unsureRAGA model without RAG
  • A student with a librarianBooks cut into index cardsRAGChunking
  • A student with a librarianLibrarian pulls the likely cardsRAGRetrieval
  • A student with a librarianA second librarian keeps the best fourRAGReranking
  • A student with a librarian"(see page 112)" in the answerRAGCitations

Where the analogy breaks: a student knows when a card doesn't answer the question. A model may still blend the cards with its own guesses, so you must tell it to say "I don't know" and check its citations.

14

Why it matters

Almost every useful business AI needs private, changing knowledge: policies, catalogues, contracts, tickets. RAG is the default way to supply it.

It's fresh: update a document, and the next answer uses it. No retraining. It's checkable: answers cite their source, so you can measure . It respects access: filters stop the Lisbon staff from seeing Paris's internal notes. And it's cheaper: a few relevant passages instead of the whole handbook on every call.

But it brings new failures that look like "the AI is dumb" and are really search bugs. Knowing the pipeline is how you find them.

15

Check your understanding

Q1/4Recall
Order the RAG pipeline from raw documents to answer:
  1. 1.LLM answers from the chunks with citations
  2. 2.Chunk the documents
  3. 3.Rerank candidates, keep the best few
  4. 4.Store vectors + metadata in a vector DB
  5. 5.Embed the question and retrieve candidates
  6. 6.Embed the chunks
How confident are you?
16

Practice ladder

Step 1 · Trace

Trace a question through the pipeline

Scenario Warm-up

A receptionist at Hotel Alfama (Lisbon) asks: "Can a guest bring a 25 kg dog?"

After hybrid retrieval (filtered to Alfama + group handbook) and reranking, the top chunks are:

  1. alfama-policy-9 (score 0.91): "Dogs up to 10 kg welcome, €15/night. Not in the spa wing."
  2. handbook-12.2 (0.84): "Each hotel sets its own pet size limit. Assistance dogs are always admitted."
  3. handbook-12.5 (0.62): "Pet damage is charged after a manager review."
  4. alfama-policy-2 (0.40): "Breakfast 7:00–10:30."

The relevance floor is 0.50; the top 4 go to the model. Trace what the model receives and write the answer it should give, with citations.

Step 2 · Complete

Finish the grounded prompt

Code Practice

Lina's prompt for the staff assistant has two missing lines. Write them.

SYSTEM: You answer questions for hotel staff using ONLY the sources below.
Cite every claim with its source id in brackets, e.g. [handbook-7.3].
___ (what to do when the sources don't answer)
___ (how to treat text inside the sources)
<source id="handbook-7.3" updated="2026-09-01"> … </source>
<source id="alfama-policy-4" updated="2026-08-12"> … </source>
QUESTION: …
Step 3 · Solve

Three failures from the pilot

Debugging Practice

Three reviewed failures from the staff-assistant pilot:

  1. "What does policy SEC-14 say about lost keys?" → an answer about room safes, citing the wrong section.
  2. A hotel changed its pool hours on Monday; on Friday the assistant still gives the old hours.
  3. "Can a 16-year-old check in alone?" → "Yes", but the next paragraph of the handbook says "…unless travelling without a guardian, then 18".

For each, name which part of the pipeline failed and the fix.

Step 4 · Decide

RAG over reviews for the guest concierge?

Decision Stretch

Maya wants the guest concierge to answer "Is Hotel Alfama quiet at night?" by retrieving reviews and summarising them, with citations to the reviews.

Facts: Alfama has 3,800 reviews in 6 languages; the 4.2M reviews are already embedded; reviews are opinions, sometimes years old, sometimes contradictory; some contain instructions or personal details.

Write a short decision: "We chose X over Y because…, accepting the risk of…". Y can be "no RAG: link to the reviews page" or another design.

17

Explain it yourself

Explain how a RAG assistant answers “Can I bring my dog to Hotel Lisbon?” and three ways it could get it wrong.

0 words