LLM API costs: tokens × price, and the levers that cut it 10×
Friday, 16:00: the AI concierge's first week in Lisbon cost €5,040 for 14,000 conversations. Maya wants it in every city next month, which at this rate means €650k a month, twice StayHub's whole cloud bill.
AI features are billed by the word, in and out. Learn where the money goes and the levers that can cut the bill around ten times.
- Problem
A hotel support assistant works beautifully in the demo. In production it handles 20,000 requests a day and the first invoice is a five-figure surprise.
- Why the simple system fails
Teams estimate with 'a few cents per call', send the whole conversation and every tool definition each turn, and route every question to the biggest model.
- Engineering concept
Cost = requests × (input tokens × input price + output tokens × output price). Every lever attacks one factor: fewer calls, fewer or cheaper input tokens, fewer output tokens, a cheaper model.
- Technology
Token usage logging, prompt caching of stable prefixes, model routing through an AI gateway, batch APIs for offline work, semantic caches, history summarisation.
- Trade-offs
Smaller models and shorter context save money but can lower quality; caches can serve stale or wrong answers; batch is cheap but slow. Measure quality alongside cost.
The problem
2 min- Maya · Product manager14,000 conversations in a week. Guests love it. I want every city next month.
- Karim · CTOIt cost €5,040 for one city. All cities is about 60,000 a day: €650k a month. Our whole cloud bill is €310k.
- You · Engineer at StayHubCan't we just switch to a cheaper model?
The big idea
4 minCost = calls × (input tokens × price + output tokens × price): find the biggest factor and attack it, without losing quality.
Picture a translator paid per word read. Hand over the same 40-page contract with every question, and you pay for 40 pages each time. A translator who remembers them charges an "already read" rate.
How it works
5 min- 1Every call resends the prefix
A is the unit models read and bill, about ¾ of a word. Each call sends input tokens (a 6,000-token prefix: , 30 tools, hotel policies; history; a 100-token message) and gets output tokens back (~300).
- 2History grows every turn
Turn 1: 6,100 in. Turn 2: 6,500. Turn 8: 8,900. Total: 60,000 in (€0.30), 2,400 out (€0.06). 48,000 of the input is the same prefix, sent 8 times.
- 3Lever 1: prompt caching
bills a byte-identical prefix from a recent call at ~10%. Stable content first, changing content last: €0.36 → ≈ €0.10.
- 4Lever 2: model routing
sends each request to the cheapest model that handles it well. Breakfast hours to a small model, cancellations to the large one. 75% small: ≈ €0.028.
- 5Other levers
Fewer input tokens (summarise old turns), shorter replies (), fewer calls ( for generic questions), cheaper timing ( for offline work).
- 6Quality gate
Every lever must keep the 82% resolution rate on the 400-conversation . A human support chat costs ~€4.
Watch a decision being made
4 minYou bring the concierge from €0.36 to €0.028
Before rolling out to 60,000 conversations a day, cut cost at least 10× without lowering the 82% resolution rate. You lead; Lina checks.
- 1Log tokens on every call
Record input, cached and output tokens, model, cost, feature and city through the AI gateway.
Why: Without per-feature token logs, you're guessing which lever matters.
- 2Make the prefix cacheable
Move anything that changes (today's date, the guest's name) out of the top and into the last message. Order: system prompt, tools, policies, history, new message. ≈ €0.10.
text[system prompt][30 tools][Lisbon policies] <- identical every call, cached [history so far][new message + date + guest name] <- changesWhy: Caching matches from the first byte. One changing line at the top and nothing is cached.
- 3Route easy questions
Small model for FAQs; large model for cancellations, rebookings, complaints. 0.25 × €0.103 + 0.75 × €0.0036 ≈ €0.028.
Why: Most concierge traffic is easy. Large-model prices for "what time is breakfast?" is the expensive habit.
- 4Prove quality held
Run the 400-conversation eval on old and new setups; ship each lever behind a flag; keep it only if resolution stays ≥ 82%.
Why: A cheaper setup that resolves fewer conversations sends guests to €4 human chats.
In real life
3 minDuolingo: a premium tier
- Launched Duolingo Max in 2023, a higher-priced subscription with GPT-4-powered features
- Only paying users trigger the expensive calls
Fits when the AI feature is a premium extra.
Klarna: free for everyone, replacing support
- Publicly said in 2024 its AI assistant handled about two thirds of customer service chats in its first month
- Paid for by fewer human support conversations
Fits when each AI conversation replaces something costlier.
What goes wrong
3 minA timestamp in line one makes every prompt unique; nothing is cached.
Stable content first, changing content last.
History is resent every turn.
Model whole conversations and track cost per conversation.
"What's my booking reference?" must never return another guest's answer.
Cache only generic answers, scoped per hotel, with expiry.
Trade-offs
3 min| Option | Fits when | You pay |
|---|---|---|
| One large model for everything | Prototypes, low volume, every request hard | Highest cost; grows in step with traffic |
| Caching + routing | High volume, mixed easy and hard, an eval set | A router to maintain; misrouting risk |
| Batch API | Offline work: summaries, tagging, nightly evals | Results in hours, not seconds |
- Maya · Product managerLaunch every city on the large model only. First impressions matter. We optimise later.
- Karim · CTO"Later" is €650k a month. Caching and routing from day one.
- Maya · Product managerAnd if the router sends a furious guest to the small model in week one?
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The quality bar any cut must keep: 82% resolution on the 400-conversation set.
- The budget per conversation before rollout: you set the target (around €0.03), not the agent.
- Which requests may go to the small model (): FAQs yes, rebookings with refunds no.
The concierge costs €0.36 per conversation; we want about 13x less before rolling out. Plan first, in order. One: reorder prompt building so the stable 6,000-token prefix (system prompt, 30 tools, hotel policies) comes first with nothing variable before it, then show me cached input tokens from the usage logs. Two: propose a router sending simple questions to the small model. Log tokens and cost per call. Don't change the prompt's wording or tools. After each step, run the 400-conversation eval and report cost and resolution rate.
- Cached input tokens in the usage logs are well above zero. Zero means something variable is still at the start.
- Each eval report shows cost and resolution; any step below 82% is rejected.
- Router examples show rebookings and refunds staying on the large model.
- It moves everything to the small model and reports a saving without running the eval.
- It puts the current time or guest name at the top of the prompt.
- It shortens the system prompt or drops tools to save tokens without asking.
Check yourself
2 minNext episode: the team starts building StayHub's own concierge, and on day one it confidently tells a guest about a rooftop pool that doesn't exist.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You bring the concierge from €0.36 to €0.028
The concierge costs €0.36 per conversation in Lisbon. Before rolling out to 60,000 conversations a day, bring it down at least 10× without lowering the 82% resolution rate on the 400-conversation eval set. You lead; Lina checks.
- 1Log tokens on every call
Route every call through the AI gateway and record input, cached and output tokens, model, cost, feature and city.
tsmetrics.record("llm_call", { feature: "concierge", city: "lisbon", model, inputTokens, cachedInputTokens, outputTokens, costEur: costOf(model, usage), });Why: Providers return token counts on every response. Without logging them per feature, you're guessing which lever matters.
- 2Find the biggest factor
Per conversation: 60,000 input tokens (€0.30) and 2,400 output (€0.06). 48,000 of the input tokens are the same 6,000-token prefix, sent 8 times.
Why: The formula has several factors. Attacking a small one, like trimming replies first, would save little.
- 3Make the prefix cacheable
Move anything that changes (today's date, the guest's name) out of the system prompt and into the last message. Stable content first: system prompt, tools, policies, then history, then the new message. Each call re-reads everything up to the previous call from the cache; only the last reply and the new message (400 tokens a turn) are full price. Cached input is billed at ~10%: €0.36 → ≈ €0.10.
Why: Caching matches the prompt from its first byte. One changing line at the top makes every call unique, and nothing is cached.
- 4Route easy questions
Breakfast hours, Wi-Fi, directions: small model. Cancellations, rebookings, complaints: large model. With 75% on the small model (≈ €0.0036 each): 0.25 × 0.103 + 0.75 × 0.0036 ≈ €0.028.
Why: Most concierge traffic is easy. Paying large-model prices for "what time is breakfast?" is the expensive habit.
- 5Prove quality held
Run the 400-conversation eval set on old and new setups. Ship each lever behind a flag; keep it only if resolution stays ≥ 82%.
Why: A cheaper setup that resolves fewer conversations pushes guests to human support at ~€4 each, which costs far more than the tokens saved.
| Lever | Attacks | Typical saving | Risk |
|---|---|---|---|
| Prompt caching | Price of the repeated prefix | Large when a long stable prefix is resent | Only works if the prefix is identical and comes first; caches expire after minutes |
| Model routing | Price per token | Often 50–80% if most traffic is easy | Hard questions misrouted to the small model; needs evals per route |
| Shorter context | Input tokens | Grows with conversation length | Summaries lose details (an allergy, a date) |
| Concise output | Output tokens (the pricier meter) | Meaningful when replies are long | Answers too terse to help |
| Semantic cache | Number of calls | Proportional to repeated questions | Stale or wrong-guest answers if keyed carelessly |
| Batch API | Price, for offline work | Often around half price (illustrative) | Results in hours, not seconds |
Interactive simulation
Visual diagram
A hotel support assistant with every cost lever in place. Follow one request and see which lever applies where.
Analogy
- A translator paid by the wordPay per word readAn LLM APIInput tokens
- A translator paid by the wordPay more per word writtenAn LLM APIOutput tokens
- A translator paid by the wordHanding over the whole contract each timeAn LLM APIResending long history
- A translator paid by the wordDiscount for pages already readAn LLM APIPrompt caching
- A translator paid by the wordJunior translator for simple questionsAn LLM APIRouting to a small model
- A translator paid by the wordOvernight pile at a lower rateAn LLM APIBatch API
Where the analogy breaks: a human translator remembers yesterday's work. A model forgets everything between calls, unless you send it again and pay for it again.
Check your understanding
Practice ladder
Trace a 4-turn conversation
Estimation Warm-upA guest has a 4-turn conversation with the concierge on the large model, no caching.
- Prefix: 6,000 tokens. Each guest message: 100. Each reply: 300.
- Large model: €5 per 1M input tokens, €25 per 1M output tokens.
Trace the input tokens for each turn, then the cost of the conversation.
Fix the prompt order
Code PracticeA teammate reports that caching is on, but the gateway shows 0% cached input tokens. Here's how the prompt is built. Reorder it so the cache can work, and explain which line broke it.
const prompt = [
`Today is ${new Date().toISOString()}. Guest: ${guest.name}.`,
SYSTEM_PROMPT, // 1,500 tokens, same for everyone
TOOL_DEFINITIONS, // 3,000 tokens, same for everyone
hotelPolicies(cityId), // 1,500 tokens, same per city
...history,
newMessage,
];Summarise 50,000 reviews a night
Estimation PracticeMaya wants a short summary of new guest reviews for every hotel page. Each night: 50,000 reviews, about 400 tokens each in, 80 tokens out per summary. A teammate plans to call the large model live, one review at a time, when each review arrives.
Prices: large €5 / €25 per 1M; small €0.20 / €0.80; batch API at 50% off (illustrative). Summaries only need to be ready by the next morning.
Estimate the monthly cost of the teammate's plan and propose a cheaper one, with numbers and one quality check.
A semantic cache for the concierge?
Decision StretchAbout 20% of concierge questions are near-repeats per hotel: "Is breakfast included?", "What's the Wi-Fi password?", "Can I check in early?". A teammate proposes a semantic cache: if a new question is similar to one asked before, return the stored answer without calling the model.
Concerns raised: early check-in depends on tonight's occupancy; the Wi-Fi password changes monthly; some questions are about the guest's own booking.
Write a short decision: "We chose X over Y because…, accepting the risk of…", including what you would and wouldn't cache.
Explain it yourself
Explain to a product manager why the AI concierge costs 10× what the demo suggested, and how you'd bring it down without making it worse.