Episode 148 · A cheaper model and a smarter one

LLM quality vs cost: how much intelligence does each request need?

Tuesday, 10:00. In the same week, the model provider ships a cheaper model and a smarter one at twice the price. Booking changes cost StayHub 0.35 € each today, 6,000 a day. Maya wants the smart one, Karim the cheap one. It's your call.

Big AI models are smarter but slower and pricier. Here's how to spend that intelligence only where it pays off.

30 min advancedLLM quality vs cost vs latencyModel routingModel cascades (try a cheap model, escalate on low confidence)Cost per successful taskLatency vs model intelligenceDefining "good enough" with evals

Problem
StayHub's concierge handles ~6,000 booking changes a day in chat. The large model resolves 92% of them but costs 15× more per task than the small one, and its p95 of 4.8 s sits just under the 5-second chat limit. The small model is fast and cheap but resolves only 70%.
Why the simple system fails
One model for everything either overpays for easy questions or under-serves hard ones; choosing by price per token ignores failures, retries and humans cleaning up mistakes.
Engineering concept
Treat intelligence as a resource to allocate per task: define "good enough" with evals, route or cascade to the cheapest model that meets it, cache what repeats, and measure cost per successful task.
Technology
Model tiers, an AI gateway with routing and cascades, prompt and response caching, golden datasets and evals, per-task cost tracking.
Trade-offs
Routing and cascades cut cost and latency but add components, failure modes and eval work; a single big model is simpler and often right at low volume or for high-stakes tasks.

The problem

2 min

Tuesday, 10:00. Booking changes in chat: ~6,000 a day, at ~0.35 € each today.

Maya · Product manager
The new large-2 resolves 96% of booking changes. Switch.
Karim · CTO
Small-2 costs a fifteenth of today's model per task. Switch to that.
Lina · Senior engineer, your mentor
You lead this one. I'll only check your answer. Guess first.

The big idea

4 min

Price per token vs cost per success

The big idea

Allocate intelligence per task: pick the cheapest setup that clears your quality and latency bars, judged by cost per successful task.

Price per token is only an input. The business pays for outcomes: a failed booking change still costs its model calls, then a support agent. So neither Maya nor Karim is right as stated.

How it works

5 min

Think of a hotel front desk: trainees handle most requests, a supervisor checks anything unusual, the manager steps in when needed. What matters isn't anyone's hourly rate; it's the cost of getting each guest's problem solved.

Choosing a model setup, in order

1
Set the bars first
The pass bar is the share of tasks that must be resolved correctly, set from what an error costs: here ≥ 93%. The latency bar is how long the channel can wait: chat p95 ≤ 5 s, voice well under 1 s, batch hours. Agree both before seeing any results.
2
Measure on your own tasks
Run every candidate on your : here 400 real booking-change conversations. Public leaderboards don't test your tools or your guests' phrasing.
3
Price one task, all in
Cost per successful task = model calls + checks + 4 € × handoffs, divided by the tasks done right. Small ~1.20 €, small-2 ~0.60 €, large ~0.35 €, large-2 ~0.22 €.
4
Filter, then minimise
Drop every setup that misses a bar, then take the cheapest left. Small-2 misses 93%; large-2's p95 of 6.5 s misses the chat bar.
5
Allocate per request
A router sends obvious hard intents to the big model. A tries a cheap model first, runs a check, and escalates only on doubt. Our cascade: ~0.21 €, 95%, p95 4.6 s.
6
Keep it in CI
Golden set, bars and selection script run on every model or prompt change. Next quarter's models get the same test in an afternoon.

Watch a decision being made

4 min

Worked example

You choose the setup for booking changes (Lina only checks)

Two new models shipped the same week. Booking changes cost ~0.35 € each today (~63,000 € a month). Pick a setup Maya and Karim can both check, and make the decision repeatable.

1Define good enough
Pass bar ≥ 93% (a wrong change costs a refund and a support call). Chat latency bar p95 ≤ 5 s.
Why: Without bars set in advance, whoever has a favourite model picks the metric.
2Price every candidate all in
Run old small, small-2, large, large-2 and a small-2 → large cascade on the 400-conversation golden set; count model calls, checks and 4 € per handoff.
python
```
eligible = [c for c in candidates if c["pass_rate"] >= 0.93 and c["p95"] <= 5.0]
choice = min(eligible, key=lambda c: c["cost_per_success"])
```
Why: Counting failures reverses the price-per-token ranking.
3Apply the bars, then choose
Large-2 fails latency; small-2 fails the pass bar; today's large sits at 92%, just under our own bar. Only the cascade passes both: ~0.21 €, 95%, 4.6 s.
Why: Requirements filter; cost decides among what's left.
4Check the check, then automate
Measure the cascade's check on labelled data: false accepts (wrong answers passed) and false rejects. Put the eval script in CI. Large-2 goes to email requests, after its own eval.
Why: A cascade is only as cheap as its check is honest; every false accept ships a wrong change to a guest.

In real life

3 min

Two well-known products, opposite ends

Inline code completion (e.g. GitHub Copilot)

Suggestions must appear while you type
Favours fast models and short outputs
A late suggestion is useless, however good

Speed is part of quality.

Deep research agents (e.g. ChatGPT deep research)

Publicly described as taking several minutes per report
Spends many model calls on one answer
Users start it and come back later

Depth is worth the wait.

What goes wrong

3 min

Comparing models by price per token

Failures cost retries and humans; the cheapest model can be the most expensive setup.

Instead

Compare cost per successful task, including human fallback.

Cascading with an unmeasured check

If the check accepts wrong answers, the cascade looks cheap and ships errors.

Instead

Measure its false-accept and false-reject rates on labelled data.

Caching personal answers

A serves one guest's answer to anyone asking something similar.

Instead

Cache only generic, per-hotel answers; key by hotel and intent; expire on data changes.

Trade-offs

3 min

Setup	Fits when	You pay
One large model	Low volume, high stakes, early on	Highest cost; slow for chat and voice
Router	High volume, easy and hard intents easy to tell apart	Misrouted hard requests; more evals
Cascade	Outputs checkable automatically (tool calls, schemas)	Escalations pay twice; only as good as the check

Wednesday, 15:00. Next: booking changes on the phone concierge.

Maya · Product manager
Phone callers are the stressed ones. Give voice large-2: 96% resolved.
Omar · On-call SRE
Its first token is over 2 seconds. That's dead air after every sentence. Callers hang up.
Maya · Product manager
We can say "let me check that" while it thinks.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The bars, before any results: ≥ 93% resolved on the golden set and chat p95 ≤ 5.0 s.
The unit you compare: cost per successful task with the ~4 € human fallback, never price per token.
Which answers may come from a cache: generic hotel facts only, never anything about a guest's own stay.

Tell Claude Code

Plan first, then build an eval runner in `evals/booking-changes/` that runs our 400-conversation golden set against: large today, small-2, large-2, and a small-2 → check → large cascade. For each, report resolved rate, p95 latency, model cost per task, and cost per successful task counting 4 EUR per human handoff. Report the check's false-accept rate. Mark setups below 93% or above 5.0 s p95 as out. Don't change production routing.

Check before you accept

Recompute one row by hand: cascade ≈ 0.0075 € model cost + 5% × 4 € ≈ 0.21 € per task.
Setups that fail a bar are marked out, even when they're cheapest.
Same golden set for every setup, not edited to help one of them.
Turn per-task numbers into a month (× 180,000) and compare with today's ~63,000 €.

Red flags

The cost table leaves out the human fallback, so small-2 looks cheapest.
The cascade's check is the small model grading its own answer.
A semantic cache added "to save more", including personal questions.

Check yourself

2 min

Q1/3Reasoning

A small model costs €0.002 per booking-change task and resolves 70% of them. Every unresolved task goes to a human agent at €4. What is the total cost per task, in euros?

€

How confident are you?

Next season, Maya starts building features with Claude Code herself, and the very first one breaks a board used by 3,000 hotels.

Finished?

Mark it complete to update your skill map.

Next: How AI coding agents work (and why "done" is not proof)

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

The cheapest setup that clears the bars

Worked example

You choose the model setup for booking changes (Lina only checks)

Two new models shipped the same week. Booking changes in chat cost ~0.35 € each today (~63,000 €/month). Decide the setup for booking changes, with numbers Maya and Karim can both check, and a way to repeat the decision next time.

1Define "good enough" first
Pass bar: ≥ 93% resolved correctly (a wrong change costs a refund and a support call). Latency bar for chat: p95 ≤ 5 s.
Why: Without bars set in advance, whoever has the favourite model picks the metric. The bars come from the cost of errors and the channel, not from a leaderboard.
2Measure every candidate on your own tasks
Run old small, small-2, large, large-2 and a small-2 → large cascade on the 400-conversation golden set for booking changes. Record resolved rate, all model calls, handoffs and latency.
Why: Public benchmarks don't test your tools, your hotels or your guests' phrasing.
3Compute cost per successful task
Model calls + checks + 4 € × handoffs, divided by the tasks that end resolved (by the AI or by the person it hands off to: here, all of them). Old small ~1.20 €, small-2 ~0.60 €, large ~0.35 €, large-2 ~0.22 €, cascade ~0.21 €.
python
```
cost = model_cost + check_cost + HUMAN_COST * handed_off
cost_per_success = total_cost / n   # every task ends resolved: by the AI or by a person
```
Why: This is the number that includes what failures cost. It reverses the price-per-token ranking.
4Apply the bars, then minimise
Eligible for chat: large (92%… just under the bar, kept only as today's baseline) and the cascade (95%, 4.6 s). Large-2 fails latency; small-2 alone fails the pass bar. Choose the cascade.
Why: Requirements filter; cost decides among what's left. Lina: "And you noticed today's setup doesn't actually clear your own bar. Good."
5Check the check
Measure the cascade's check on labelled data: false accepts (wrong answers passed) and false rejects (good answers escalated).
Why: A cascade is only as good as its check. If it accepts wrong changes, the low cost is an illusion paid by guests.
6Put the decision in CI
The golden set, the bars and the eligibility script run on every prompt or model change. Large-2 goes to email booking requests, where latency doesn't matter, after its own eval.
Why: Models improve and reprice every quarter. With the script in CI, the next decision takes an afternoon.

const HIGH_STAKES = new Set(["refund", "complaint", "group_booking"]);

export async function bookingChange(req: GuestRequest) {
  const tier = HIGH_STAKES.has(req.intent) ? "large" : "small2";   // route obvious cases
  let reply = await callModel(tier, req);
  if (tier === "small2" && !checkToolCall(reply, req)) {            // cascade on doubt
    reply = await callModel("large", req);
  }
  if (reply.needsHuman) await handoffToAgent(req, reply);           // the most expensive tier
  return track(req, reply, { tier: reply.tier, cost: reply.totalCost, resolved: !reply.needsHuman });
}

Route obvious high-stakes intents, cascade the rest, and record cost and outcome per task, so cost per success can be measured in production too.

Interactive simulation

Visual diagram

Guest

AI gateway + router

Response cache

Small model

Confidence check

Large model

Human agent

Evals + cost tracking

Click a component

A guest assistant that spends intelligence where it changes the outcome. Follow the walkthrough.

Analogy

A hotel front deskAn AI product

A hotel front deskA trainee answers routine questionsAn AI productA small, cheap model
A hotel front deskA printed FAQ cardAn AI productA cache of common answers
A hotel front desk"Unsure? Call the duty manager"An AI productA cascade to the large model
A hotel front deskThe manager's time: costly, slow to getAn AI productLarge model: pricier and slower
A hotel front deskCost of solving each guest's problemAn AI productCost per successful task

Where the analogy breaks: a trainee learns on the job. A model doesn't improve by itself, but new, cheaper models arrive every few months. Teams with good evals can switch to them safely.

Check your understanding

Q1/3Decision

Which task is the best candidate for a large, slow model via a batch API at a discount?

How confident are you?

Practice ladder

Step 1 · Trace

Cost per task for the email classifier

Estimation Warm-up

StayHub sorts ~10,000 guest emails a day into 12 categories. A misrouted email costs ~0.50 € (a support agent re-routes it). Results on the golden set:

Setup	Model cost / email	Correct
Small-2	0.0004 €	97%
Large	0.006 €	98.5%

Trace the cost per email for each, and the monthly difference. Which would you choose?

Step 2 · Complete

Finish the eligibility script

Code Practice

Complete Lina's summary and selection code so it picks the cheapest setup that clears the bars.

HUMAN_COST = 4.00
PASS_BAR = 0.93
LATENCY_P95_BAR = 5.0

def summarize(name, results):
    n = len(results)
    passed = sum(r.passed for r in results)
    model_cost = sum(r.model_cost for r in results)
    human_cost = ___                                   # (1)
    p95 = sorted(r.latency for r in results)[int(0.95 * (n - 1))]
    return {"setup": name, "pass_rate": passed / n, "p95_latency": p95,
            "cost_per_success": ___}                   # (2)

eligible = [c for c in candidates if ___]            # (3)
choice = min(eligible, key=lambda c: c["cost_per_success"])

Step 3 · Solve

Allocate intelligence across four tasks

Design Practice

For each StayHub task, choose a setup (small model, large model, router, cascade, cache, batch) and a latency bar, and say why in one or two sentences:

Breakfast, parking and wifi questions in chat (~14,000/day).
Nightly summaries of guest reviews for revenue managers (~40,000 reviews/night).
Refund requests in chat (~900/day; policy engine decides the money).
Voice concierge questions about hotel facilities (~8,000 calls/day).

Step 4 · Decide

Next quarter: switch the cascade's escalation to large-2?

Decision Stretch

Three months later, the provider makes large-2 faster: p95 now 3.9 s in chat, same price (0.060 € per call). On the golden set for booking changes:

Setup	Total cost / task	Resolved	p95
Cascade small-2 → large (current)	0.21 €	95%	4.6 s
Cascade small-2 → large-2	0.13 €	97%	4.2 s
Large-2 alone	0.22 €	96%	3.9 s

Write a short decision: "We chose X over Y because…, accepting the risk of…", and say what you'd check before switching.

Explain it yourself

Your CFO says: "The small model is 25× cheaper. Switch everything to it." Explain your answer.

0 words