Episode 141 · Paying a genius to read the breakfast menu

Routing for cost: small models for most, large models where it matters

Monday, 9:30. Traces show 70% of the Plus Assistant's 20,000 daily questions are "what time is breakfast?" and "is parking free?". StayHub pays the large model about 10,140 € a month to answer every one of them.

Most questions are easy. Send them to a cheap model, save the expensive one for the hard ones, and check you didn't break anything.

30 min advancedRouting strategies: heuristics, classifiers and cascadesModel cascades (try a cheap model, escalate on low confidence)Guarding routing quality with evals and monitoringBlended cost and savings of a routing splitModel routingLLM quality vs cost vs latency

Problem
StayHub's Plus Assistant costs about €10,140/month on a large model, even with prompt caching. Most questions are "what time is breakfast?" and "is parking free?"; a few are "rebook my family of five across two properties with a loyalty voucher".
Why the simple system fails
Using the large model for everything overpays massively for easy questions. Switching everything to a small model saves money but quietly ruins the hard conversations, which are often the most valuable ones.
Engineering concept
Route each request to the cheapest model that can handle it, using cheap signals, a classifier, or a try-small-then-escalate cascade, and prove quality per route with evals.
Technology
AI gateway routing rules, small classifiers or embedding-based routers, confidence and validation checks for cascades, golden datasets, shadow traffic and canary rollouts.
Trade-offs
Savings of 50–80% in exchange for a new component that can misroute, extra latency on escalations, and an eval suite you must keep current.

The problem

2 min

Monday, 9:30. The Plus Assistant costs about 10,140 € a month with caching.

Karim · CTO
Traces say 70% of the 20,000 daily questions are routine: breakfast, parking, wifi. We pay the large model for all of them.
You · Engineer at StayHub
The small model costs about a twentieth. Switch everything?
Lina · Senior engineer, your mentor
You lead this one. What happens to the other 30%?

The big idea

4 min

Not every question needs the big model

The big idea

Send each request to the cheapest model that can handle it, and prove it per route with evals.

A hotel doesn't send the duty manager to answer "what's the wifi password?". The front desk handles routine questions, and some topics always go to the manager.

How it works

5 min

chooses a model per request. A is a tiny model that labels a request easy or hard with a probability. A tries small first and escalates if the answer fails a check.

One request through the router

1
Pinned intents
A pinned intent always goes large: refunds, rebookings, group bookings, complaints, whatever the classifier says.
2
Free signals
FAQ widget → small. Attachments → large. No model call needed.
3
Classifier for free text
Only the guest's short message is classified; "easy" at p ≥ 0.85 goes small.
4
Log the reason
Every route records why (pinned, faq-widget, clf:0.91), so misroutes can be traced.
5
Score per route
Eval scores and handoffs per route and per bucket, not one average.

Watch a decision being made

4 min

Worked example

You ship the router (Lina checks)

Cut ~10,140 €/month without hurting refunds, rebookings and complaints. You have 600 labelled conversations and traces with UI intents.

1Use the signals you already have
Pin high-risk intents; FAQ widget to small. These cover about half the traffic before any classifier.
Why: Rules cost nothing, add no latency and are easy to explain.

2Classify only the free text

A tiny classifier on the message, not the full context:

const { label, p } = await classify(req.message);
if (label === "easy" && p >= 0.85) return { model: "small", reason: `clf:${p}` };
return { model: "large", reason: `clf-${label}` };

Why: Reading the whole context would cost a real share of the small call.

3Prove it per bucket
Golden set routed vs all-large: ship only if each bucket stays within 0.02.
Why: Hard items are a small share; a collapse there hides inside a good average.
4Shadow, then canary
Log decisions without acting for a week, then route 5% of hotels, watching scores, handoffs and cost.
Why: Real traffic differs from the golden set.

In real life

3 min

Who chooses the model?

ChatGPT with GPT-5: an automatic router

OpenAI described a router picking a fast or a deeper model per request
When it misjudges, users get a weaker answer without knowing why

Fits users who shouldn't think about models.

GitHub Copilot Chat: the user picks

Lets developers choose the model from a list
Experts trade speed, cost and quality themselves

Fits expert users.

What goes wrong

3 min

Routing by message length alone

"Cancel everything" is short and high-stakes.

Instead

Combine signals and pin high-risk intents.

A cascade with a weak check

If the check is the small model's own confidence, bad answers pass.

Instead

Cascade only machine-checkable outputs: schemas, totals, tests.

Setting the threshold once

Traffic, prompts and models change.

Instead

Re-run routing evals on every change; alert on handoffs per route.

Trade-offs

3 min

Strategy	Per request	Monthly
All large	0.0169 €	~10,140 €
80% small / 20% large	≈ 0.0041 €	~2,460 €
Cascade, 25% escalate	≈ 0.0051 €	~3,075 €
95% small	≈ 0.0017 €	~1,020 €

Friday, 10:00. Threshold back at 0.85, group bookings pinned.

Karim · CTO
If per-bucket evals hold, aim for 95% small: 1,440 € a month saved.
Maya · Product manager
The last 15% are the messy ones: accessibility needs, unhappy guests. That's where our reviews come from.
Karim · CTO
That's 17,000 € a year.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The target split (~80/20) and the intents pinned to large: group rebookings, refunds, complaints.
The quality bar per bucket and per intent, never the average.
How a threshold rolls out: offline evals, shadow mode, a 5% canary with a rollback trigger.

Tell Claude Code

Add cost routing to the Plus Assistant in `services/ai-gateway/routes/plus-assistant.ts` (20,000 requests a day, ~10,140 EUR a month, ~70% routine). Plan first. Heuristic rules first; a small classifier for free text; group rebookings, refunds, complaints pinned large; log route, reason, cost. Report golden-set scores per bucket and per intent, then add a shadow mode. Don't change live routing until I've seen per-bucket results.

Check before you accept

The hard bucket and pinned intents score as well as on the large model.
Blended math: ≈ 0.0041 € per request, ~2,460 € a month, large route ~82% of it.
Pinned intents are a rule before the classifier, not a hint to it.

Red flags

Threshold tuned until the average stays flat while small passes 90%.
No per-bucket or per-intent breakdown.
Escalation based on the model's own "I'm confident" line.

Check yourself

2 min

Q1/3Reasoning

A large-model request costs €0.025 (no caching) and a small-model request €0.00092. With 600,000 requests/month routed 70% small / 30% large, what's the monthly cost in euros?

€

How confident are you?

Next episode: Maya arrives with a feature request from five hotel owners, and before anyone writes code, Lina asks: what problem are we solving?

Finished?

Mark it complete to update your skill map.

Next: Problem first: start from the user, not the technology

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Rules, then a classifier, then proof

Worked example

You ship the Plus Assistant router (Lina checks)

Cut the Plus Assistant's ~10,140 €/month without hurting refunds, rebookings and complaints. You have 600 labelled conversations and the traces with UI intents.

1Size the prize by route
From traces: 70% routine, 30% other. Price both routes: small ~0.0009 €, large ~0.0169 €. At 80/20: ~2,460 €/month.
Why: Knowing the large share is ~82% of the routed bill tells you where the next euro comes from. Lina: "And where it doesn't."
2Use the signals you already have
Pin refund, rebooking, group-booking and complaint intents to large; FAQ widget to small; attachments to large.
Why: These rules cost nothing, add no latency and are easy to explain. They cover about half the traffic before any classifier runs.
3Classify only the free text
A tiny classifier on the guest's message (not the full context) labels easy/hard; route small only for "easy" at p ≥ 0.85.
ts
```
const { label, p } = await classify(req.message);
if (label === "easy" && p >= 0.85) return { model: "small", reason: `clf:${p}` };
return { model: "large", reason: `clf-${label}` };
```
Why: A classifier reading the whole context would cost a real share of the small call and add latency.
4Prove it per bucket
Run the golden set routed and all-large. Ship only if each bucket (easy, medium, hard) stays within 0.02, and the hard bucket rarely goes small.
Why: Hard items are a small share; a collapse there hides inside a good average.
5Shadow, then canary
Log routing decisions without acting on them for a week (shadow), then route for 5% of hotels (canary), watching scores, handoffs and cost per route.
Why: Real traffic differs from the golden set. Shadow mode shows the mix; the canary limits who's affected if it's wrong.

Strategy	Extra cost per request	Extra latency	Main failure mode	Best for
Heuristics	~0	~0	Blunt: rules misjudge free text	Intent already known from the UI or endpoint
Classifier	Small	Tens to a few hundred ms	Misroutes hard items as easy	Free text with labelled history
Cascade	Small call on every request	Full small call on escalations	Weak check lets bad answers through	Machine-checkable outputs
Hybrid	Varies	Varies	More moving parts	High volume, mixed traffic, mature evals

None is best in general; the right one depends on your signals and how checkable your outputs are.

Interactive simulation

Visual diagram

Guest

Assistant backend

Gateway + router

Small model

Large model

Answer check

Evals + metrics

Click a component

A router in the AI gateway sends most traffic to the small model, escalates on failure, and is audited by evals.

Analogy

A hospitalModel routing

A hospitalSigns at the entrance: desk A or BModel routingSimple rules
A hospitalA triage nurse's quick checkModel routingA routing classifier
A hospitalSee the GP, referred if neededModel routingA cascade: small first, then large
A hospitalAudits of patients sent homeModel routingEvals on each route

Where the analogy breaks: a sent-home patient often comes back and complains. A user given a wrong AI answer often just leaves. That's why you must measure, not wait for complaints.

Check your understanding

Q1/3Decision

You already route 80% small / 20% large and spend ~€3,440/month before any caching, of which ~87% is the large-model route. What is usually the best next lever?

How confident are you?

Practice ladder

Step 1 · Trace

Route five requests

Scenario Warm-up

The router, in order: 1. Attachment → large 2. UI intent in {refund, rebooking, group-booking, complaint} → large (pinned) 3. UI intent = faq → small 4. Classifier: "easy" with p ≥ 0.85 → small; otherwise → large

Trace these: - A: FAQ widget, "Is the pool heated?" - B: free text, "What time is check-out?" → classifier easy, 0.97 - C: free text, "I need to move my sister's booking too, she's in the room next to mine" → classifier easy, 0.88 - D: UI intent refund, "refund pls" - E: free text with a photo of a broken lamp → classifier easy, 0.95

For each, give the route and the logged reason. Is any of them a worrying route?

Step 2 · Complete

Finish the blended-cost table

Estimation Practice

Small model ≈ 0.0009 € per request, large ≈ 0.0169 €. 600,000 requests a month.

Strategy	Cost per request	Monthly
All large	0.0169 €	~10,140 €
80% small / 20% large	≈ 0.0041 €	~2,460 €
90% small / 10% large	___	___
Cascade, 25% escalate	___	___

Fill in the two rows, then say which strategy you'd prefer for the Plus Assistant and why.

Step 3 · Solve

Route the review-reply drafts

Design Practice

Hotels use StayHub to draft replies to guest reviews: 8,000 drafts a day, all on the large model today. A manager reads every draft before posting.

About 75% of reviews are 4–5 stars and short ("Lovely stay, great breakfast!").
About 10% are 1–2 stars, often long, sometimes mentioning hygiene, safety or discrimination.
The UI knows the star rating.

Design the routing and how you'd prove it's safe.

Step 4 · Decide

Cascade or classifier for invoice extraction?

Decision Stretch

StayHub's invoice extraction (9,000 invoices a month) runs scans on the large document model. Business checks (sums, IBAN checksum, vendor master) already decide what goes straight through.

Small document model: about 1/8 of the large model's cost per page.
On the golden set, the small model passes all business checks on 72% of scans; the large model on 81%.
Invoices are processed in the background; nobody waits on them.

Write a short decision: "We chose X over Y because…, accepting the risk of…". Options: a cascade (small first, large if checks fail), or a classifier predicting which scans are hard.

Explain it yourself

Your CFO asks: "Why don't we just use the cheap model for everything?" Explain routing, the savings math, and the risks.

0 words