Episode 110 · PMS-417, not PMS-502

Hybrid search and reranking

Thursday, 6:40 am: a receptionist with a queue of guests asks what kiosk error PMS-417 means, and the assistant confidently explains PMS-502. Questions with codes, room numbers or hotel names find the right passage only 41% of the time.

Keyword search finds exact codes, vector search finds meaning. Use both to collect candidates, then a careful reranker to put the best one first.

30 min advancedHybrid searchRerankingReciprocal rank fusion (merging ranked lists)Two-stage retrieval: cheap recall, expensive precisionRetrieval recall vs precisionSemantic search

Problem
A guest asks “What does kiosk error PMS-417 mean?” and your RAG bot confidently explains PMS-502 instead.
Why the simple system fails
Pure vector search ranks by meaning. Every kiosk-error chunk means roughly the same thing, so the exact code that matters gets no special weight; and stuffing more chunks into the prompt only adds noise and cost.
Engineering concept
Retrieve in two stages: a cheap, wide first stage that maximises recall (keyword + vector, fused by rank), then an expensive, narrow second stage that maximises precision (a reranker reading query and passage together).
Technology
BM25 over an inverted index, an ANN vector index, reciprocal rank fusion (RRF), a cross-encoder reranker, and retrieval metrics like recall@k, MRR and nDCG.
Trade-offs
Every stage adds latency, infrastructure and tuning knobs. Hybrid search costs a second index; reranking costs 50–300 ms and GPU or API spend per query. Measure before you add either.

The problem

2 min

Thursday, 6:40 am. A queue of guests at a self check-in kiosk in Lisbon.

Omar · On-call SRE
"What does kiosk error PMS-417 mean?" The assistant explained PMS-502 and said to restart the key encoder. Wrong fix, longer queue.
Omar · On-call SRE
Questions with codes, room numbers or hotel names find the right chunk in the top 4 only 41% of the time.
Lina · Senior engineer, your mentor
You lead the diagnosis. Guess first.

The big idea

4 min

Two searches that fail in opposite ways

The big idea

Retrieve in two stages: stage 1 is for recall (don't miss the right chunk), stage 2 is for precision (put it first).

Keyword search nails exact tokens and misses synonyms; vector search gets meaning and blurs codes. They fail in opposite ways, so you run both, then let a slower, smarter model order the shortlist.

How it works

5 min

Two-stage retrieval for "kiosk error PMS-417"

1
Keyword search
BM25 scores passages by how often the question's words appear, weighting rare words more (). "PMS-417" is rare, so the one chunk containing it scores highest.
2
Vector search
finds meaning: "the kiosk won't print a key", phrased any way, in any language.
3
Fuse by rank
Hybrid search runs both and merges them (). RRF () gives each passage 1 / (60 + rank) per list, summed. Only ranks count, so BM25 and cosine scores never need comparing.
4
Rerank the shortlist
A cross-encoder reads the question and one passage together and scores relevance (). Accurate but slow, so it only sees the top 50.
5
Send the best 4
Only the top 4 (~2,000 tokens) go to the model, with citations.
6
Measure each stage
for stage 1 (did we find it?), recall@4 for stage 2 (is it on top?). Code questions: 41% vector-only → 79% hybrid → 91% with the reranker.

Watch a decision being made

4 min

Worked example

Lina checks the two-stage retriever, step by step

Bring code and name questions from 41% to 90%+ recall@4 without sending more tokens to the model.

1Label the failures
"Add 120 code and name questions to the eval set, each with its correct chunk. Vector-only on them: 41%."
Why: "Search feels bad" can't be fixed; "41% on code questions" can, and tells you which stage to add.
2Add keyword search, fuse by rank
"Postgres full-text on the same chunks, same hotel filter, top 50, in parallel with vector top 50. Merge with RRF." Recall@50: 96%.
ts
```
const [kw, vec] = await Promise.all([
  bm25Search(q, { filter, limit: 50 }),
  vectorSearch(await embed(q), { filter, limit: 50 }),
]);
const candidates = rrf([kw, vec]).slice(0, 50);
```
Why: The first stage must find the chunk; a reranker can't rank what it never sees.
3Rerank the 50
"Found, but often ranked 10–30. The cross-encoder keeps the top 4." Overall recall@4: 93%. p95 retrieval: 40 → 220 ms.
Why: Only a model reading question and passage together reliably orders near-identical chunks.
4Check the bill
"About €900 a month via API, and the context stays at 4 chunks. 220 ms is fine for staff."
Why: Each stage must pay for itself; sending 30 chunks would cost far more.

In real life

3 min

Rent a reranker, or run one

Cohere Rerank: a hosted API

Send the query and candidates, get relevance scores back
No GPUs to run; pay per search
Another vendor sees your passages; network time on every call

Fits moderate volume and teams without GPU operations.

Open cross-encoders, self-hosted

Widely used open models (for example from the sentence-transformers ecosystem) on your own GPUs
Data stays in-house; fixed cost, cheap at high volume
You run, scale and update it

Fits high volume, strict data rules, or a platform team.

What goes wrong

3 min

Vector-only search for codes, numbers and names

Embeddings blur exact tokens.

Instead

Add BM25 and fuse with vector results.

Adding raw BM25 and cosine scores together

Different scales that shift with every model change.

Instead

Fuse by rank with RRF.

Reranking only a short list, or adding stages blind

The reranker can't rescue a chunk it never sees; untested stages add latency for nothing.

Instead

Retrieve wide (30–50), then rerank; track recall@50 and recall@4 per question type.

Trade-offs

3 min

Retrieval	Choose when	You pay
Vector only	Descriptive questions, no exact identifiers	Codes, numbers and names blurred
Hybrid (BM25 + vector, RRF)	Users mix exact lookups and descriptions	Two indexes; rough order in the shortlist
Hybrid + reranker	The right chunk is found but ranked too low	+50–300 ms; reranker cost or GPUs

Friday, 15:00. Should the guest concierge get the reranker too?

Maya · Product manager
For guests, every 200 ms shows in the chat. Hybrid alone is 86%. Skip the reranker.
Omar · On-call SRE
A guest told the wrong wifi code calls the front desk anyway. 93% beats 86%.
Maya · Product manager
Guests ask "is breakfast included?", not kiosk codes.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Whether to add keyword search, based on the eval: vector-only recall@4 is 41% on code and name questions.
Whether a reranker is worth it: 86% → 93% recall@4, for p95 40 → 220 ms and about €900 a month.
Which numbers you track per stage: recall@50, recall@4, latency.

Tell Claude Code

The staff assistant answered kiosk error PMS-417 with PMS-502's explanation. Add BM25 keyword search next to pgvector, both with the same hotel and language filters (top 50 each), merge with reciprocal rank fusion (k = 60), rerank with a cross-encoder and send the top 4. Keep the current pipeline behind a flag. Don't send more chunks. Plan first, then run the eval (including the 120 code questions) and report recall@50, recall@4 and p95 per variant.

Check before you accept

Ask the PMS-417 question: its chunk is in the top 4 and cited.
The 120 code and name questions are reported separately; an overall number can hide them.
Filters apply to both searches, not only the vector one.

Red flags

Misses "fixed" by sending 30 chunks to the model.
A reranker added while recall@50 is low.
Raw BM25 and cosine scores added together.

Check yourself

2 min

Q1/3Debugging

Your vector-only RAG answers questions about error code E-2041 with the explanation for E-2014. Rephrasing the prompt doesn't help. What's the most targeted fix?

How confident are you?

Next episode: a guest asks if the Paris hotel's spa is open on Mondays. The assistant says yes, 9:00–21:00, with a citation. The spa closed for renovation two weeks ago.

Finished?

Mark it complete to update your skill map.

Next: Citations, grounding and knowing when to say “I don't know”

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Fusion by rank, then a reranker

Reciprocal rank fusion: for each passage d, RRF(d) = Σ 1 / (k + rank_L(d)) over every list L it appears in, with k commonly 60. k damps the gap between rank 1 and rank 2, so one list can't dominate, and only ranks are used, so keyword and vector scores never need to be on the same scale.

Worked example

You build the two-stage retriever

Bring questions with codes, room numbers and hotel names from 41% to 90%+ recall@4 without sending more tokens to the model. Lina checks your numbers at the end.

1Label the failures
Add 120 code and name questions to the eval set, each with its correct chunk. Vector-only recall@4 on them: 41%.
Why: "Search feels bad" can't be fixed; "41% on code questions" can, and tells you which stage to add.
2Add keyword search
Postgres full-text (BM25-like) on the same chunks, same hotel filter, top 50, run in parallel with vector top 50.
Why: Rare exact tokens like PMS-417 are exactly what keyword scoring rewards and embeddings blur.

3Fuse by rank

Merge with RRF (k = 60). Recall@50: 96%; recall@4 on code questions: 79%.

const [kw, vec] = await Promise.all([
  bm25Search(q, { filter, limit: 50 }),
  vectorSearch(await embed(q), { filter, limit: 50 }),
]);
const candidates = rrf([kw, vec]).slice(0, 50);

Why: Ranks need no score calibration, and they don't break when the embedding model changes.

4Rerank the shortlist
A cross-encoder scores the 50 candidates with the question; keep the top 4. Code questions: 91%; overall 93%. Retrieval p95: 40 → 220 ms.
Why: The right chunk was found but often ranked 10–30. Only a model reading question and passage together can reliably order near-identical chunks.
5Check cost and latency
Reranker ≈ €900/month via API; context stays at 4 chunks (~2,000 tokens). Lina: "Checked. And 220 ms is fine for staff; we'd stream for guests."
Why: Each stage must pay for itself. Here the alternative, sending 30 chunks, would cost far more.

Metric	Question it answers	Use it for
Recall@k ()	Is the right chunk anywhere in the top k?	Stage 1 (k = 50): did we miss it?
MRR ()	How high is the first right chunk, on average?	Stage 2: is it near the top?
nDCG ()	Are the most relevant chunks ranked first, with graded relevance?	Comparing rerankers
Answer accuracy	Is the final answer correct and cited?	The whole pipeline

Measure each stage with the metric that matches its job.

Interactive simulation

Visual diagram

Front-desk staff

RAG service

BM25 index

Vector index

RRF fusion

Cross-encoder reranker

LLM

Click a component

Two-stage retrieval for StayHub's staff assistant. Click any component, or follow the walkthrough.

Analogy

Hiring for a jobHybrid search and reranking

Hiring for a jobRecruiter scanning CVs for exact keywordsHybrid search and rerankingKeyword search (BM25)
Hiring for a jobRecruiter reading for overall fitHybrid search and rerankingVector search
Hiring for a jobMerging both shortlistsHybrid search and rerankingReciprocal rank fusion
Hiring for a jobA 30-minute interview with the top 20Hybrid search and rerankingA cross-encoder reranker
Hiring for a jobCan't interview all 5,000 applicantsHybrid search and rerankingReranking only the shortlist

Where the analogy breaks: a great interviewer might find a gem the recruiters missed. A reranker can't. It only reorders what stage 1 found, so a miss there is final.

Why it matters

Search quality is a ceiling on answer quality. If the right chunk never reaches the model, it can't answer correctly. At best it refuses. At worst, it makes something up, a .

Learn the symptoms that point at search, not the model. Codes, IDs or hotel names come back wrong: that's vector-only search. The right hotel's policy loses to another hotel's: that's a ranking problem. Sending more chunks "fixes" it, but costs more and gets noisier: you're paying the AI model to do a reranker's job.

Spot the pattern, and you fix the right stage instead of swapping models or rewriting prompts.

Check your understanding

Q1/3Reasoning

With RRF and k = 60, a chunk is ranked 2nd by BM25 and 3rd by vector search. What is its fused score? (Ranks are 1-based.)

How confident are you?

Practice ladder

Step 1 · Trace

Fuse two lists by hand

Estimation Warm-up

For "kiosk error PMS-417", the two searches return (rank: chunk):

Keyword: 1: kiosk-417, 2: kiosk-147, 3: handbook-kiosk-intro
Vector: 1: kiosk-502, 2: kiosk-412, 3: kiosk-417, 4: handbook-kiosk-intro

Compute the RRF score (k = 60) for each chunk and give the fused order. Which chunk comes first, and why?

Step 2 · Complete

Finish the retriever

Code Practice

Lina's two-stage retriever has three gaps. Fill them in.

export async function retrieve(question: string, user: StaffUser) {
  const filter = ___;                       // (1) from who is asking
  const [kw, vec] = await Promise.all([
    bm25Search(question, { filter, limit: 50 }),
    vectorSearch(await embed(question), { filter, limit: 50 }),
  ]);
  const candidates = ___;                   // (2) fuse, keep 50
  const ranked = await rerank(question, candidates);
  return ___;                               // (3) what goes to the model
}

Step 3 · Solve

Room numbers and hotel names

Design Practice

Hotel managers get a new search over maintenance tickets (2M tickets, short texts like "Room 1204: AC dripping, tray replaced"). They search things like "room 1204 leaks", "anything about the Alfama boiler", "AC problems last summer in Porto".

Design the retrieval: which searches, how to merge them, filters, and how you'd show it works.

Step 4 · Decide

Rent or run the reranker for guests?

Decision Stretch

The guest concierge will make about 180,000 searches a day (60,000 conversations × 3). Options for its reranker:

API: ~€0.001 per search, ~100 ms of network time, passages leave StayHub (some contain guests' booking details).
Self-hosted: 2 GPUs ≈ €4,500/month plus ~20% of an engineer (≈ €2,000/month); ~40 ms; data stays in-house.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a teammate why your RAG pipeline has two retrievers and a reranker instead of just a bigger top-k, and how you'd know if any of them is unnecessary.

0 words