Episode 131 · The same question, 14,000 times a day

Prompt caching, response caching and semantic caching

Wednesday, 11:00: 38% of concierge questions are near-duplicates ("what time is breakfast?"), and each one still pays for a full model call and 3.2 seconds of waiting. And the provider's prompt cache has never hit once.

"Caching" means three different things in AI apps. Each one saves more money than the last, and each one is riskier.

30 min advancedPrompt caching to cut repeated input costPrefix stability for prompt cachingExact-match LLM response cachingSemantic caching and its risksLLM cache key designCache invalidation and TTLs

Problem
StayHub's concierge sends the same ~2,500-token system prompt, tool definitions and hotel policies with every message, and 38% of guest questions are near-duplicates of a few hundred FAQs. You pay full price for all of it, every time.
Why the simple system fails
A normal HTTP cache keyed on the exact request almost never hits: every conversation has different history, wording and timestamps. Naively caching answers by 'similar meaning' serves one guest's answer to another, or yesterday's price today.
Engineering concept
Cache at the right layer: reuse the provider's computed prefix (prompt caching), reuse whole answers only for identical deterministic requests (response cache), and reuse answers for similar questions only where the answer is truly shared and stable (semantic cache).
Technology
Provider prompt caching with a stable prefix layout; a key-value store for exact responses; an embedding model plus vector index with a similarity threshold for semantic caching; TTLs and versioned cache keys.
Trade-offs
Prompt caching is nearly free and safe but only cuts input cost. Response and semantic caches skip the model entirely but risk stale, leaked or wrong-but-similar answers.

The problem

2 min

Wednesday, 11:00. The gateway's new dashboard is up.

Maya · Product manager
38% of concierge questions are near-duplicates: breakfast, parking, check-in.
Omar · On-call SRE
Each pays a full model call, 3.2 s p50. And the provider's prompt-cache hit rate is 0%.
Lina · Senior engineer, your mentor
You propose; I'll check.

The big idea

4 min

Three caches, three layers

The big idea

Cache what repeats, at the layer where it repeats, and share only answers that are the same for everyone.

Prompt caching changes nothing about answers, so ship it today. A semantic cache skips the model and can serve a wrong answer, so pilot it.

How it works

5 min

Prompt caching: the provider reuses its work on a recent prefix; the model still answers. A response cache returns a stored answer for an identical request. A semantic cache returns a stored answer for a question with the same meaning, found by embedding similarity.

Caching an AI assistant, layer by layer

1
Stable prefix first
System prompt, tools, hotel policies at the top, byte-identical. Timestamp, guest details and the new message last ().
2
Decide what's cacheable
Shared and stable (breakfast hours): yes. Personal (my booking), live (rooms tonight) or actions: never.
3
Build the key
A cache key holds everything that changes the answer: model, prompt version, hotel, language, question ().
4
Semantic lookup
Filter by scope, then take the closest question above a high (0.95), plus a check on negations and numbers ().
5
Expire and purge
24 h ; purge a hotel's entries when it edits its policies.
6
Measure wrong hits
Sample 200 hits a week and check each answer fits its question.

Watch a decision being made

4 min

Worked example

You add caching to the concierge (Lina checks)

Cut cost and latency for repeated questions without ever serving a guest an answer meant for someone or somewhere else.

1Find what repeats
The 2,500-token prefix repeats every call; ~38% of questions match FAQs; almost no full request repeats exactly.
Why: Each kind of repetition has its own layer. An exact cache would almost never hit here.
2Fix the prefix
Reorder: system prompt, tools, hotel policies, conversation, volatile context last. A call drops from ~0.025 to ~0.014 EUR.
Why: One changing byte early invalidates everything after it.
3Classify cacheability
Only shared-faq questions may use the semantic cache. "Is breakfast included?" is personal.
Why: Identical text, different answer per booking: cacheability is decided by meaning, not similarity.
4Scope and check every hit
Filter by hotel, locale, prompt version, age under 24 h.
ts
```
if (hit && hit.similarity >= 0.95 && sameNegationsAndNumbers(q, hit.question)) return hit.answer;
```
Why: Scope keeps hotels apart; the cheap check catches "not" and "for 4", which embeddings blur.

In real life

3 min

Two providers, two ways to cache prompts

Anthropic: explicit cache breakpoints

Publicly documented: you mark where the cacheable prefix ends
Cache reads billed at a fraction of input; writes cost a bit more

More control; you mark the prefix yourself.

OpenAI: automatic prompt caching

Publicly documented as automatic for long-enough prompts
Still only hits when the prefix is identical

Zero effort, if your prompt is laid out for it.

What goes wrong

3 min

Volatile data at the top of the prompt

A timestamp on line 1 makes every prefix unique.

Instead

Stable first, volatile last.

Semantic-caching personal or live questions

"Is breakfast included?" differs per booking; "rooms tonight?" changes by the minute.

Instead

Classify cacheability first; shared, stable FAQs only.

Trusting similarity near the threshold

"Pets allowed" and "pets not allowed" can score above 0.95.

Instead

A negation/number check and sampled review of hits.

Trade-offs

3 min

Cache	Buys	Risk
Prompt cache	Cheaper input, faster first token	None to answers; input only
Exact response cache	Zero model cost on hits	Rare hits in chat; incomplete keys leak
Semantic cache	Biggest savings on FAQs	Wrong-but-similar answers

Thursday, 15:00. Reviewing the pilot plan.

Karim · CTO
38% repeat. Cache all concierge answers by meaning, minus an exclusion list, and take the full saving.
Omar · On-call SRE
Azur's dogs were one missing key field.
Lina · Senior engineer, your mentor
You designed it. Hold your ground or change it.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The prompt order: stable parts first, volatile values like the date and guest name last.
Which questions may use a shared semantic cache: FAQ-classified, non-personal only.
The key (hotel, locale, prompt version), threshold (0.95), TTL (24 h), what purges it, and the wrong-hit rate you'll act on.

Tell Claude Code

Two caching changes in `src/concierge/`, each its own commit. Plan first. One: move the stable prefix (system prompt, tools, hotel policies) first and the timestamp and guest name last; show cached-token share before and after on 100 replayed requests. Two: a semantic cache for FAQ-classified questions only, keyed by hotel_id, locale, prompt_version; threshold 0.95, TTL 24 h, purged on policy edits; reject hits differing in a negation or number. Test the pets question at Mira and Azur.

Check before you accept

Cached-token share jumps from 0% to most of the input, with the same answers.
hotel_id, locale and prompt_version are all in the key.
Three "my booking" questions never hit the cache, and a policy edit stops the old answer immediately.

Red flags

Threshold lowered "to improve the hit rate" without measuring wrong hits.
Answers built from a guest's booking stored in the cache.
"Saves 38%" claimed from FAQ share instead of measured hits.

Check yourself

2 min

Q1/3Debugging

You enabled prompt caching, but your cached-token share is 0%. The prompt is 6,000 tokens and starts with: Current time: 2026-10-02T09:14:07Z. You are the assistant for Hotel Aurora…. What is the most likely cause?

How confident are you?

Next episode: Karim has a quote for GPUs and wants to run an open-weight model ourselves. You lead the decision.

Finished?

Mark it complete to update your skill map.

Next: API models vs self-hosted open models

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Prefix first, then a scoped semantic cache

Worked example

You and Lina add caching to the concierge

Cut the concierge's cost and latency for repeated questions without ever serving a guest an answer meant for someone or somewhere else.

1Find what repeats
From a week of gateway logs: the 2,500-token prefix repeats on every call; ~38% of questions match a few hundred FAQs; almost no full request repeats exactly (every chat has its own history).
Why: Each kind of repetition has its own cache layer. An exact HTTP cache would almost never hit here.
2Fix the prefix
Reorder: system prompt → tools (sorted deterministically) → hotel policies (breakpoint) → conversation → volatile context last. Track the prompt-cache hit rate per route.
Why: One changing byte early in the prompt invalidates everything after it. Hit rate is the proof the layout works.
3Decide cacheability
A small classifier labels each question: shared-faq (breakfast hours, parking), personal (my booking), live (rooms tonight), action (book this). Only shared-faq may use the semantic cache.
Why: "Is breakfast included?" is identical text with a different answer per booking. Cacheability is decided by meaning, not by similarity.
4Scope the key and threshold
Filter by hotel, locale, prompt version and age < 24 h; accept the nearest match only if similarity ≥ 0.95 and the negation/number check passes.
ts
```
if (hit && hit.similarity >= 0.95 && sameNegationsAndNumbers(q, hit.question)) return hit.answer;
const answer = await callModel(req);
if (cls === "shared-faq") await cache.put({ hotelId, locale, promptVersion, question: q, answer });
```
Why: Scope keeps hotels apart; a high threshold plus a cheap check catches "not" and "for 4" that embeddings blur.
5Expire and purge
24 h TTL; when a hotel edits its policies, delete its entries; a new prompt version starts a fresh set.
Why: Stale answers are wrong answers with good latency. Purges make a hotel's edit visible to guests immediately.
6Measure wrong hits
Each week, sample 200 cache hits; a calibrated judge (from the evaluations module) and a human spot-check whether the cached answer fits the new question.
Why: Hit rate and savings always look good. Only the wrong-hit rate tells you if the cache is hurting guests.

Question pair	Typical similarity	Same answer?
"What time is breakfast?" / "When do you serve breakfast?"	Very high	Yes (same hotel)
"Are pets allowed?" / "Are pets not allowed?"	Very high	Opposite intent
"Room for 2 on Friday?" / "Room for 4 on Saturday?"	High	No, and availability changes by the minute
"Is breakfast included?" (guest A) / same text (guest B)	Identical	Depends on each booking

Embeddings capture topic well, and small decisive words (not, numbers, dates, "my") poorly.

Interactive simulation

Visual diagram

Guest

Assistant service

Exact response cache

Embedding model

Semantic cache index

AI gateway

LLM provider (prompt cache)

Booking DB

Click a component

Where each cache sits in the request path. Start the walkthrough to follow a cache hit and a miss.

Analogy

A hotel conciergeAI caching

A hotel conciergeHas memorised the handbookAI cachingPrompt caching
A hotel conciergeChange page one, memory is uselessAI cachingOpening text must match exactly
A hotel conciergeA printed card: "Wi-Fi password?"AI cachingExact response caching
A hotel concierge"I've heard that before" without listeningAI cachingSemantic caching
A hotel conciergePromising late checkout to the wrong guestAI cachingA confident wrong cache hit

Where the analogy breaks: a concierge can notice a guest is different. A semantic cache only compares meanings, so it can't tell "my booking" from "your booking" unless you separate them.

Check your understanding

Q1/3Scenario

Input price is €5 per million tokens and cached input is billed at 10% of that. A request has 3,000 input tokens, of which 2,500 hit the prompt cache. What is the input cost of this request in euros?

€

How confident are you?

Practice ladder

Step 1 · Trace

Trace four questions through the caches

Scenario Warm-up

The concierge's semantic cache: only questions classified shared-faq; lookup filtered by hotel, locale, prompt version, age < 24 h; accept if similarity ≥ 0.95 and negations/numbers match. Prefix caching is on for every call.

Cached entry: Hotel Mira, en, prompt v12, 10 h old: "What time is breakfast?" → "7:00–10:30 every day."

Trace each question (all at Hotel Mira, English, prompt v12 unless noted): 1. "When do you serve breakfast?" (similarity 0.97) 2. "Is breakfast included in my stay?" (similarity 0.91) 3. "What time is breakfast?" at Hotel Azur 4. "What time is breakfast?" after Mira edited its policies 1 hour ago

Step 2 · Complete

Reorder the prompt

Code Practice

Here's the concierge's current prompt, in order. Rewrite the order so the provider's prompt cache can hit, mark where you'd put the cache breakpoint, and say which items must never appear before it.

1. "Today is {now}. Guest: {name}, tier {tier}."
2. System prompt (v12)
3. Booking summary for this guest
4. Tool definitions (v7)
5. Hotel policies for {hotel}
6. Conversation so far
7. The guest's new message

Step 3 · Solve

Caching for the inbox triage

Design Practice

The inbox assistant (from the security module) classifies 6,000 emails a day into intents with a fixed 1,800-token instruction block, on the small model, at temperature 0. About 15% of emails are exact duplicates (newsletters, automated booking confirmations from partners). Design its caching: which layers, the keys, and anything you must NOT cache.

Step 4 · Decide

Raise the threshold or widen the cache?

Decision Stretch

Pilot results after 4 weeks (5 hotels, FAQ-only): hit rate 31% of FAQ questions at those hotels, answered in ~40 ms instead of ~3 s. Weekly review of 200 hits found a 2.5% wrong-hit rate; most wrong hits were number mismatches ("parking for 2 cars" answered with the 1-car price). The agreed bar was 1%.

Options: (A) raise the threshold to 0.97; (B) keep 0.95 and strengthen the number check; (C) roll out to all hotels as is. Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain the difference between prompt caching and semantic caching, and when semantic caching becomes dangerous.

0 words