Episode 109 · The exception in the next chunk

Chunking strategies: cutting documents so retrieval works

Tuesday, 11:00: a receptionist asks whether a 16-year-old can check in alone, and the assistant says yes. The handbook's very next paragraph says 18 without a guardian, but it landed in a different chunk.

Search returns pieces of documents, not whole documents. How you cut those pieces decides whether the AI can ever find the answer.

30 min advancedChunkingChunk size: precision vs contextStructure-aware and semantic chunkingChunk overlapChunk metadata and contextual headersMetadata filtering in retrieval

Problem
RAG can only answer from what it retrieves, and it retrieves chunks. If the answer is split across two chunks, buried in a huge one, or stripped of the context that says which hotel it applies to, the model can't use it.
Why the simple system fails
Splitting every document into fixed 1,000-character pieces cuts sentences and tables in half, mixes unrelated sections, separates a fact from its subject ('the fee is €25' about what?), and produces chunks that match everything weakly.
Engineering concept
Chunk along the document's own structure, at a size that matches the questions you expect, with enough overlap or parent context to keep facts whole, and enrich each chunk with metadata and a header so it stands on its own.
Technology
Recursive splitters, Markdown/HTML/PDF layout parsers, semantic (embedding-breakpoint) chunking, overlap, parent-document retrieval, contextual chunk headers, metadata filters, retrieval evals (recall@k).
Trade-offs
Small chunks match precisely but lose context; large chunks keep context but dilute the embedding and cost tokens. Smarter chunking costs ingestion time and complexity. There is no universal best size, only the best for your documents and questions, measured.

The problem

2 min

Tuesday, 11:00. A receptionist's question from last night.

Omar · On-call SRE
"Can a 16-year-old check in alone?" The assistant: "Yes, from age 16 [handbook-41]".
Maya · Product manager
The very next paragraph: "Without a guardian, the minimum age is 18." The citation was real. The answer was wrong.
You · Engineer at StayHub
So the prompt needs a rule about exceptions?
Lina · Senior engineer, your mentor
Guess first. Then look at what the model actually received.

The big idea

4 min

The chunk is the unit of meaning

The big idea

The chunk is the unit of meaning: a good chunk stands on its own, with a clear subject and a complete fact.

The test: hand a chunk alone to a new colleague. Can they tell what it's about and answer from it? If not, neither can the model, however good it is. No prompt rule fixes an exception the model never saw.

How it works

5 min

cuts documents into passages; each is embedded as one vector and retrieved alone. Our v1 cut every 1,000 characters, with no overlap and no headers. Here are the better options, and what to add.

Ways to cut a document, and what to add

1
Fixed size
Every N characters or tokens. Simple, and blind to sentences, rules and tables. Our v1.
2
Recursive and structure-aware
: split on sections first, then paragraphs, then sentences, only as far as needed. Structure-aware: use the document's own headings and tables, keep a rule with its exception and a table whole.
3
Semantic
: cut where the topic shifts. Useful for transcripts with no structure.
4
Overlap and headers
Overlap repeats a sentence across a boundary so a fact that straddles it survives (). A contextual header like "Hotel Alfama › Policies › Parking" is embedded with the chunk, so "The fee is €18" says what it's about ().
5
Size: match small, send big
Small chunks match precisely but lose context; big ones keep context but blur the match. Small-to-big () matches 250-token children and sends their ~1,200-token parent section.
6
Measure recall
Recall@k is how often the passage that answers a question is among the top k retrieved. Measure it on real questions before every re-index.

Watch a decision being made

4 min

Worked example

Lina re-chunks the handbook and hotel pages

Raise recall@4 on the 300-question staff eval set from 61% without blowing up the context size.

1Build the eval set first
"300 real staff questions, each with the answering passage marked. That's how we'll know a change helped."
Why: Chunking failures are invisible in demos. Without an eval set, every change is a guess.
2Parse the structure
"Extract headings, lists and tables from the PDFs, not raw lines. Fee tables stay whole."
Why: A table split in two separates "Pets" from "€15 per night".
3Split by section, add context
"One chunk per section when it fits (~500 tokens), else paragraphs, with one sentence of overlap. Prefix each with its path; store hotel, section and date."
text
```
StayHub group handbook › 4 Check-in › 4.2 Minimum age

Guests may check in from age 16. If travelling without
a guardian, the minimum age is 18. ID is required.
```
Why: Section 4.2 now carries the rule and its exception together, and says what it's about.
4Measure, then consider small-to-big
Recall@4: 61% → 84%. Small-to-big reaches 90%, but sends ~3,600 tokens per question instead of 2,000.
Why: Every chunking change means re-embedding everything, so test it on the eval set before the whole index.

In real life

3 min

Giving chunks the context they lack

Anthropic: contextual retrieval

Published in 2024: before embedding, add a short chunk-specific note on where it sits in its document
Reported large drops in failed retrievals, especially with keyword search and reranking
One extra model call per chunk at indexing

Fits questions answered by one passage that lacks its own context.

Microsoft: GraphRAG

Published in 2024: build a graph of entities and relationships, with summaries of related groups
Aimed at questions about a whole collection ("main themes?")
Much heavier to build and update

Fits broad questions across many documents.

What goes wrong

3 min

Fixed-size chunks on structured documents

Cuts rules from exceptions and tables in half.

Instead

Split along headings, recursively, and keep tables whole.

Chunks without their subject

"The fee is €18" matches every fee question at every hotel.

Instead

Contextual headers and metadata (hotel, section, date).

Changing chunking without an eval set or version

Demo questions hide regressions; mixed old and new chunks break citations.

Instead

Measure recall@k on real questions; re-index into a new version and switch when complete.

Trade-offs

3 min

Chunking	Good for	You pay
Small chunks	Short facts: FAQs, fee tables	Lost context; rules split from exceptions
Large chunks	Rarely the matching unit; better as what you send	Blurry vectors, more tokens per answer
Small-to-big	Long sections where rules and exceptions travel together	More tokens; two levels to keep in sync

Wednesday, 10:00. Deciding whether to ship small-to-big.

Omar · On-call SRE
Small-to-big gets recall to 90%. Fewer wrong answers at 2 am. Ship it everywhere.
Karim · CTO
3,600 tokens instead of 2,000: about €7k a month more. 84% is good.
Omar · On-call SRE
The misses are exactly the rules with exceptions. Those are the expensive mistakes.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

How documents are cut: along headings and sections, never mid-rule; a rule and its exception stay together.
What stays whole: fee tables (or each piece repeats the header row), FAQ question-and-answer pairs.
The bar a new chunker must clear on the 300-question eval (recall@4: 61% today), and the token cost you accept for it.

Tell Claude Code

Our chunker cuts every document into 1,000-character pieces with no overlap; recall@4 on our 300 marked staff questions is 61%. Build a v2 chunker next to v1: parse PDFs into headings, paragraphs and whole tables, split by section (~500 tokens, recursive fallback, one sentence of overlap), prepend "Hotel › Section" before embedding. Index into a separate versioned table; don't touch the live index. Report recall@4, chunk count and tokens per question for both.

Check before you accept

Print the chunks for §4.2: "from age 16" and "18 without a guardian" are in the same chunk.
Read three random chunks alone: you can tell the hotel and section from each header.
v1 and v2 are compared on the same 300 questions, with tokens per question.

Red flags

Bigger chunks "to be safe" with no measurement (we tried: 52% recall, ~12,000 tokens).
A new chunker judged by reading a few answers instead of the retrieval eval.
The live index overwritten in place, with no version to switch back to.

Check yourself

2 min

Q1/3Debugging

The assistant answers "Yes, dogs are welcome" but never mentions the €25 fee, though it's in the same policy, two sentences later. Chunks are 60 tokens, no overlap, top-3. What's the most targeted fix?

How confident are you?

Next episode: a receptionist asks what kiosk error PMS-417 means, and the assistant confidently explains PMS-502 instead.

Finished?

Mark it complete to update your skill map.

Next: Hybrid search and reranking

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Structure, overlap, headers, then measure

Worked example

You re-chunk the handbook and hotel pages

Raise retrieval recall@4 on the 300-question staff eval set from 61% without blowing up the context size. Lina asks a question at each step.

1Build the eval set first
300 real staff questions, each with the passage that answers it marked. Lina: "How will you know a change helped?" You: by recall@4 on this set, before and after.
Why: Chunking failures are invisible in demos. Without an eval set, every change is a guess.
2Parse the structure
Extract headings, lists and tables from the PDFs instead of raw lines. Fee tables stay whole as one chunk, rendered as Markdown rows.
Why: A table split across chunks separates "Pets" from "€15 per night"; structure is where the meaning lives.
3Split by section, recursively
One chunk per section when it fits (~500 tokens); otherwise split by paragraph, then sentence, with a one-sentence overlap.
Why: Section 4.2 now carries the rule and its exception together, and overlap rescues facts that still straddle a boundary.

4Add headers and metadata

Prefix each chunk with its path and store hotel_id, doc_type, section_path, updated_at.

const header = [doc.title, ...section.path].join(" › ");
const text = `${header}\n\n${chunk.text}`;          // embedded together
await store({ text, vector: await embed(text),
  hotelId: doc.hotelId, sectionPath: section.path, updatedAt: doc.updatedAt });

Why: "The fee is €18" now says which hotel and which fee, both for the embedding and for the citation.

5Measure, then consider small-to-big
Recall@4: 61% → 84%. Then try 250-token children for matching, sending their ~1,200-token parent section: 90%, but ~3,600 instead of 2,000 tokens per question.
Why: Every chunking change needs a full re-embed, so it's tested on the eval set before rolling out to the whole index.

Strategy	How	Good for	Watch out for
Fixed size	Every N tokens, maybe with overlap	Quick start, uniform text	Splits sentences, rules from exceptions, tables
Recursive	Biggest boundary that fits, then smaller	Most prose	Still blind to tables unless parsed first
Structure-aware	Headings, sections, lists, tables	Handbooks, policies, docs	Very long sections need sub-splitting
Semantic	Cut at topic shifts (embedding distance)	Transcripts, unstructured text	More compute at indexing; harder to debug
Small-to-big ()	Match small children, send parents	Precise questions needing context	More tokens per answer

Chunk size is a tension: small chunks match precisely but lose context; large chunks keep context but blur the match.

Interactive simulation

Visual diagram

Source documents

Parser

Chunker

Enricher

Embedding model

Vector index

Parent sections

Click a component

An ingestion pipeline where chunking is a deliberate stage: parse structure, split, enrich, embed, store small chunks for matching and their parent sections for context.

Analogy

Turning a cookbook into index cardsChunking

Turning a cookbook into index cardsScissors every 20 linesChunkingFixed-size chunks
Turning a cookbook into index cardsIngredients on one card, oven heat on anotherChunkingA fact split across chunks
Turning a cookbook into index cardsA whole chapter on one cardChunkingChunks too big to match well
Turning a cookbook into index cardsOne recipe per cardChunkingStructure-aware chunking
Turning a cookbook into index cardsRecipe name at the top of each cardChunkingA contextual header
Turning a cookbook into index cardsTags like "vegetarian, 30 min"ChunkingMetadata for filtering

Where the analogy breaks: a cook flips to the next card when a recipe continues. The AI only sees the few cards search returned, so each card must stand on its own.

Why it matters

Imagine StayHub cutting its hotels' policy handbooks into equal pieces, no overlap. The chatbot goes live. The support team collects failures.

"Can I bring my dog, and what does it cost?" gets "Yes, dogs are welcome." No price: it was in the next chunk.

"What's breakfast at Paris?" gets Amsterdam's hours. Neither chunk mentioned the hotel name, so they looked identical.

The city tax table was cut between its header and its rows. And one chunk covered late checkout, parking and the spa, so it matched everything a little and nothing strongly.

None of these are model problems. They're cutting problems, made once and felt on every question afterwards.

Check your understanding

Q1/3Reasoning

A 10,000-token document is split into 500-token chunks with 100 tokens of overlap (each chunk starts 400 tokens after the previous one). About how many chunks do you get?

chunks

How confident are you?

Practice ladder

Step 1 · Trace

Trace a fixed-size cut

Scenario Warm-up

Hotel Ribeira's policy page, cut every ~100 characters with no overlap:

[1] "Parking: underground garage, 40 spaces. The fee is €18 per night. Electric car charging is availab"
[2] "le for €6 per session. Pets: dogs up to 10 kg welcome. The fee is €15 per night. Not allowed in th"
[3] "e restaurant. Late checkout: until 12:00 free; until 14:00 for €30, subject to occupancy."

A receptionist asks: "How much is the pet fee?" Trace which chunks a vector search would likely return and what can go wrong. Then show how a structure-aware split would cut this page.

Step 2 · Complete

Write the contextual headers

Code Practice

Three chunks from different documents arrive with no context. Write a header for each (document › section path), and say which metadata fields you'd store alongside.

"The fee is €18 per night, payable at check-in." (from Hotel Alfama's policy page, section Parking)
"Exceptions require written approval from the duty manager." (from the group handbook, chapter 9 Refunds, section 9.4 Goodwill gestures)
"Available 7:00–10:30, 7:30–11:00 at weekends." (from Hotel Ribeira's page, section Breakfast)

Step 3 · Solve

Chunk the call-centre transcripts

Design Practice

Maya wants staff to search past support call transcripts ("how did we handle a guest whose flight was cancelled?"). Transcripts are 10–40 minutes of speech-to-text: no headings, two speakers, topics drift, guests' names and card digits appear.

Propose a chunking plan, the metadata, and how you'd know it works.

Step 4 · Decide

Re-chunk the whole index now?

Decision Stretch

The new chunking (structure + headers) raised recall@4 from 61% to 84% on the 300-question eval set. Applying it means re-chunking and re-embedding all policy documents: about 2M chunks, ~4 hours, ~€300 at illustrative prices.

Options: A. Re-index into a new version, run the eval set on it, switch all queries at once. B. Re-chunk each document the next time it changes, so old and new chunks coexist for months.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a teammate why 'just split everything into 1,000-character chunks' produces a weak RAG system, and what you'd do instead.

0 words