Episode 130 · The 48,000 euro bill

AI gateways and model routing

Monday, 9:00: the AI bill for StayHub's EU concierge and staff tools hit 48,000 EUR last month, 70% of it simple FAQ answers sent to the biggest model. On Tuesday a provider outage took the concierge down for 47 minutes, and nobody can say which feature spends what.

Instead of every service calling AI providers directly, send all calls through one front door that picks the model, tracks the cost and handles outages.

30 min advancedAI gateway responsibilitiesModel routingRouting by task difficulty and modalityLLM provider fallbackToken rate limits and quotasLLM cost levers: routing, batching, semantic cache, shorter context

Problem
Five StayHub features call LLMs directly. The bill for the EU concierge and staff tools reached €48,000 a month, one provider outage took down the concierge, and nobody can say which feature spends what.
Why the simple system fails
Direct SDK calls scatter keys, retries, model choices and logging across services. Every request goes to the biggest model, a single provider is a single point of failure, and one runaway job can exhaust the shared quota.
Engineering concept
A single choke point for model traffic that decides which model serves each request, what happens when it fails, who may spend how much, and what gets recorded.
Technology
AI gateway (self-hosted proxy or managed service) with routing rules, a classifier or heuristics, fallback chains, token-based rate limits, virtual keys and structured logs.
Trade-offs
One more hop and one more critical component; routing mistakes silently lower quality; logs of prompts become a sensitive data store.

The problem

2 min

Monday, 9:00. Karim forwards the invoice.

Karim · CTO
EU AI bill last month: 48,000 EUR, three times spring. Which feature is it?
Omar · On-call SRE
Five features, five keys, one invoice. And Tuesday, provider A was down 47 minutes. So was the concierge.
Maya · Product manager
Evals say 70% of calls (FAQ, email classification, review summaries) do just as well on a small model.

The big idea

4 min

One door for all model traffic

The big idea

Put one door in front of all model traffic: it routes, falls back, enforces budgets and records what each feature spends.

Negotiating a discount helps, but it doesn't fix the outage or say who spends what. A gateway is the hotel switchboard for model calls: right line, a spare line, an itemised bill.

How it works

5 min

One request through the gateway

1
Identify the caller
Each feature gets a virtual key (concierge, staff agent, inbox, reviews, evals), so cost is attributed to it. Real provider keys live only in the .
2
Check the token budget
A token budget limits tokens per minute and per month for each feature. Estimate before the call; reconcile with real usage after.
3
Pick the model
Model routing sends each request to the cheapest model that handles it well: by modality, then task name, then a small difficulty classifier only where rules can't decide ().
4
Call it once
One attempt per target, with a timeout. Retries live here only, not in each feature too.
5
Fall back if needed
A fallback chain is the ordered list of other models to try. Fail over on 429, 5xx and timeouts; never on a 400, which fails everywhere ().
6
Record it
Feature, model, tokens, cost, latency, status. Prompt text redacted or not stored.

Watch a decision being made

4 min

Worked example

Lina puts a gateway in front of the providers

Cut the 48,000 EUR bill without hurting answers, survive a provider outage, and know what each feature spends.

1Attribute the traffic
Virtual keys per feature; one week of logs: feature, model, tokens, cost.
Why: You can't cut what you can't see.
2Route by the cheapest signal
Three easy tasks go to the small model by task name; only concierge chat needs a classifier.
yaml
```
- name: easy-tasks
  match: { task: [email-classify, faq-answer, review-summary] }
  target: provider-a/small-model
  fallbacks: [provider-b/small-model]
```
Why: A task name costs nothing to check; a model-based router only earns its milliseconds where rules can't decide.
3Gate every route on evals
Run each task's slice of the golden set on both models; switch only if it doesn't regress.
Why: Moving 70% of traffic is a quality change. Without evals you learn from complaints.
4Evaluate the fallback too
Provider B's model goes through the same evals; a circuit breaker skips a failing target.
Why: A fallback is a different model: tool-call formats, JSON quirks, refusals. It can "work" while quietly getting worse.

In real life

3 min

Two gateway styles

Cloudflare AI Gateway: a proxy on your own accounts

Publicly documented as sitting between your app and providers you already use
Adds logging, caching, rate limits and fallbacks; contracts stay with each provider

Central control without changing who you buy from.

OpenRouter: one API, one bill, many models

Publicly offers many providers' models through one account
Fast to try models; another company sits in your traffic

Maximum choice; a new vendor in your data path.

What goes wrong

3 min

Retrying the same provider on 429

You're over quota; immediate retries push everyone deeper.

Instead

Respect Retry-After, fail over, or queue low-priority work.

Routing or failing over without evals

Complaints are a slow quality detector; "no errors" isn't "good answers".

Instead

Eval every route and every fallback target.

Logging full prompts forever

The gateway sees every guest name; its log becomes your most sensitive database.

Instead

Metadata always; prompts redacted, sampled, short-lived.

Trade-offs

3 min

Option	Fits when	Costs
Direct SDK calls	One or two features, small spend	No routing, budgets or fallback
Self-hosted gateway	Several teams, strict data handling	You operate a critical service
Managed gateway	Need control fast; vendor terms fit	Another vendor sees prompts

Tuesday, 10:00. The routing plan.

Karim · CTO
70% to the small model saves over 30,000 EUR a month. Flip it today.
Maya · Product manager
If the small model fumbles a refund question in the concierge, we pay more than we save.
Lina · Senior engineer, your mentor
The evals cover three tasks. Concierge chat is a different question.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which tasks may go to the small model: only those your evals proved, not "everything that looks easy".
The fallback chain and its triggers: 429, 5xx, timeouts; never a 400.
Token limits and priorities, so the nightly batch can never starve the concierge, and retries owned by the gateway alone.

Tell Claude Code

Put an AI gateway in `services/ai-gateway/` in front of our five AI features (~64,000 calls a day, 48,000 EUR last month). Plan first. Virtual key per feature; FAQ, email classification and review summaries route to the small model; fallback to provider B on 429, 5xx, timeouts only; tokens-per-minute limits per feature, batch well below concierge; remove feature-level retries. Log feature, model, tokens, cost, no prompt text. Run evals on every route and fallback.

Check before you accept

Provider keys exist only in the gateway's secrets; features hold virtual keys.
Eval results per route and per fallback target, read before traffic moves.
Limits count tokens per minute, and a simulated 503 from provider A is answered via B.

Red flags

100% of traffic to the small model because the cost chart looks great, no evals.
Retries kept in the features and added to the gateway.
A 400 "context length exceeded" fails over to every provider.

Check yourself

2 min

Q1/3Scenario

Each call costs €0.025 on the large model and about €0.001 on the small model. You make 600,000 calls a month and route 70% to the small model. What is the new monthly cost in euros?

€

How confident are you?

Next episode: 38% of concierge questions are near-duplicates of each other, and each one still pays for a full model call and 3 seconds of waiting.

Finished?

Mark it complete to update your skill map.

Next: Prompt caching, response caching and semantic caching

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Routes, fallbacks and budgets

Worked example

Lina puts a gateway in front of the providers

Cut the 48,000 EUR bill without hurting answers, survive a provider outage, and know what each feature spends.

1Attribute the traffic
Give each feature a virtual key and send all calls through the gateway for one week, logging feature, model, tokens and cost.
Why: You can't cut what you can't see. The first week shows concierge chat and review summaries are 80% of spend.
2Route by the cheapest signal
Images → vision model. faq-answer, email-classify, review-summary → small model by task name. Concierge chat → a small classifier labels easy/hard.
yaml
```
- name: easy-tasks
  match: { task: [email-classify, faq-answer, review-summary] }
  target: provider-a/small-model
  fallbacks: [provider-b/small-model]
```
Why: Task names and modality cost nothing to check. A model-based router only earns its 50 ms where rules can't decide.
3Gate every route on evals
Before switching a task to the small model, run its slice of the golden set (from the evaluations module) on both models. Switch only if the slice doesn't regress.
Why: Moving 70% of traffic is a quality change. Without evals you find out from complaints.
4Build the fallback chain
For each route: primary target, then the same-size model at provider B. One attempt per target; circuit breaker opens at 50% errors over 30 s. Run the evals on provider B too.
Why: Tuesday's 47 minutes become a failover. And a fallback you haven't evaluated may "work" while quietly breaking tool calls.
5Budget in tokens
Per-key tokens-per-minute limits and monthly budgets; the review batch gets low priority. Estimate before the call, reconcile with the provider's reported usage after.
Why: Provider quotas are in tokens. A request limit lets one long-document batch starve the concierge.
6Log safely
Always log metadata (feature, model, tokens, cost, latency, status, prompt version). Store prompt text redacted, sampled and short-lived.
Why: The gateway sees every prompt. Unredacted, its log becomes the most sensitive database we own.

Routing signal	Example	Cost of the decision	Failure mode
Modality	Image or PDF attached	~0 ms	Rarely wrong
Task name	`email-classify` → small model	~0 ms	A task drifts harder over time; nobody notices
Heuristics	Input > 8k tokens, mentions "refund"	~0 ms	Brittle; easy to miss cases
Classifier model	Tiny model labels easy / hard	20–80 ms + small cost	Sends hard requests to the small model
Cascade	Small model first; escalate if validation fails	An extra call on escalations	Doubles latency for the hard tail

Route by the cheapest reliable signal first; use models to route only where rules can't decide.

Interactive simulation

Visual diagram

Guest chat service

Review-summary job

AI gateway

Router (tiny classifier)

Small model (provider A)

Large model (provider A)

Large model (provider B)

Vision model

Usage logs + cost DB

Click a component

One gateway in front of several model providers. Start the walkthrough to follow three different requests.

Analogy

Hospital triageAn AI gateway

Hospital triageSprained wrist to a nurseAn AI gatewayEasy task to a small model
Hospital triageChest pain to a cardiologistAn AI gatewayHard task to a large model
Hospital triageX-ray to radiologyAn AI gatewayPhoto to a vision model
Hospital triagePartner hospital's on-call doctorAn AI gatewayFallback to another provider
Hospital triageEvery visit recorded with its costAn AI gatewayToken and cost tracking

Where the analogy breaks: a triage nurse can examine the patient. A gateway must decide from cheap clues, like the task type, and every routing rule changes answer quality, so test it with evals.

Check your understanding

Q1/3Reasoning

Put the gateway's steps for one request in a sensible order:

1.Reconcile actual tokens and emit the usage event
2.Call the target with a timeout; fail over on retryable errors
3.Route by modality (image? audio?)
4.Route by task or difficulty
5.Verify the virtual key
6.Check token budget and TPM limit (estimate)

How confident are you?

Practice ladder

Step 1 · Trace

Trace four requests through the gateway

Scenario Warm-up

Gateway config: images → vision model; tasks faq-answer, email-classify, review-summary → small model (fallback: provider B small); concierge-chat → classifier (easy → small, hard → large; fallback: provider B large). Fail over only on 429, 5xx, timeout. Circuit breaker on provider A's large model is open right now.

Trace each request: which model serves it, and what's logged? 1. Inbox email, task email-classify. 2. Concierge: "Can I get a refund if my flight is cancelled? Booking LIS-48213." Classifier: hard. 3. Staff uploads a photo of a damaged room, task damage-report, has_image: true. 4. Concierge request with a 300k-token pasted document, provider returns 400 (context too long).

Step 2 · Complete

Finish the budget config

Code Practice

Lina's virtual-key config has gaps. Fill in the concierge and review-batch keys so the batch can never starve the concierge again, and say why each value is set that way.

keys:
  vk-concierge:
    team: guest-xp
    tpm: ___
    monthly_budget_eur: 9000
    priority: ___
  vk-review-batch:
    team: insights
    tpm: ___
    monthly_budget_eur: 800
    priority: ___
    schedule: ___
retry:
  on: [429, 500, 502, 503, timeout]
  max_attempts_per_target: ___
  respect_retry_after: true

Step 3 · Solve

Cost per feature, then a plan

Estimation Practice

After one week through the gateway, the log shows daily averages (large model ~0.025 EUR/call, small ~0.0009):

Feature	Calls/day	Model now	Eval: small model OK?
Concierge chat (EU markets)	36,000	large	85% of messages (easy ones)
Staff agent	6,000	large	no
Inbox triage	8,000	large	yes
Review summaries	6,000	large	yes
Nightly evals (judge)	8,000	large	judge must stay large

Compute today's daily cost per feature, propose a routing plan, and estimate the new monthly bill.

Step 4 · Decide

Self-host or managed gateway?

Decision Stretch

Two options for StayHub's gateway: - A: a managed gateway service: live this week, dashboards included, but it processes our prompts (guest names, booking details) on its servers; its data terms are under legal review. - B: a self-hosted open-source proxy: two weeks of platform work, we run it in our two regions with high availability.

Karim wants savings this month. Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a product manager why the team wants an AI gateway, and what could go wrong if routing is done carelessly.

0 words