Episode 140 · What one conversation really costs

The LLM cost model: from tokens to a monthly bill

Monday, 10:00. The Plus Assistant, StayHub's new WhatsApp assistant for Plus members, cost 340 € in its pilot last month. Karim must tell the board on Thursday what it costs at 33 times the traffic, and whether the 9 € StayHub Plus plan can absorb it.

How a few simple numbers turn into your monthly AI bill, so you can predict it instead of being shocked by it.

30 min intermediateMonthly LLM cost formula (requests × tokens × price)Input vs output token pricingPrompt caching to cut repeated input costContext growth makes long conversations expensiveCost per conversation and per userCost tagging and unit economics

Problem
StayHub's Plus Assistant, a new WhatsApp assistant for premium members, costs about €340 in the pilot month. Leadership asks what it will cost at full rollout and whether the €9/month premium plan can absorb it.
Why the simple system fails
Teams extrapolate from the pilot invoice or from a "price per 1M tokens" headline. They miss that output tokens cost several times more than input, that every turn re-sends the whole history, and that the system prompt is paid on every single call.
Engineering concept
Model cost as a formula: requests × (input tokens × input price + output tokens × output price), adjusted for cached input, then divide by conversations and users to get unit economics.
Technology
Token usage fields in API responses, per-request cost logging, prompt caching, cost dashboards per feature and per tenant.
Trade-offs
Cheaper per request usually means shorter context, smaller models or stale caches. Each saving has a quality, latency or freshness price that you must measure.

The problem

2 min

Monday, 10:00. The Plus Assistant, a WhatsApp assistant for StayHub Plus members, is about to roll out.

Karim · CTO
The pilot cost 340 € at 600 requests a day. Rollout is 20,000 a day, about 33×. So ~11,300 €/month, inside the 9 € Plus plan.
You · Engineer at StayHub
Same model, same feature: multiplying seems fair.
Lina · Senior engineer, your mentor
Guess what it misses.

The big idea

4 min

Why the pilot invoice lies

The big idea

Cost = requests × (input tokens × input price + output tokens × output price), minus cache discounts. Forecast from the formula, never from last month's invoice.

A chat is a taxi with no memory: each new stop replays the whole route, and you pay for every kilometre again.

How it works

5 min

Bills are per million (about ¾ of a word). Tokens the model writes cost more than tokens you send: here 25 € vs 5 € per million. discounts an identical beginning of the prompt, here to 10%.

One rollout request, priced

1
Stable prefix
Instructions, 14 tool definitions, hotel policies: about 1,800 tokens, identical every time.
2
History + message
About 1,200 tokens, growing every turn: each turn re-sends the conversation so far.
3
Reply
About 400 output tokens, priced 5× higher than input.
4
Price it
3,000 × 5 €/M = 0.015 €; 400 × 25 €/M = 0.010 €. 0.025 € per request.
5
Scale it
× 20,000 a day × 30 = 15,000 €/month, not 11,300: the pilot sent ~1,800 input tokens, rollout 3,000.
6
End on unit economics
: cost per member vs the price. 15,000 / 5,000 = 3.00 €, 33% of the 9 € plan.

Watch a decision being made

4 min

Worked example

Lina forecasts the rollout bill

Give Karim a defensible monthly number and say whether it fits the 9 € plan (target: AI cost under 30% of the price per member).

1Measure tokens, don't guess them
A week of beta spans: median 3,000 input (1,800 stable prefix), 400 output.
Why: Guests type 20 words; we send 3,000 tokens.

2Price from reported usage

Three terms per call:

return (uncached * p.inPerM
  + cached * p.inPerM * p.cachedFactor
  + output * p.outPerM) / 1_000_000;

Why: The API reports real usage; estimates drift.

3Apply caching
1,800 cached × 0.50 €/M + 1,200 × 5 €/M + output 0.010 € = 0.0169 €: ~10,140 €/month.
Why: The prefix is identical every time, so it's a discount with no quality risk.
4Check the tail
Cost per conversation at p50, p95, p99: 25-turn chats re-send history every turn.
Why: Averages hide heavy users.

In real life

3 min

Two well-known pricing choices

GitHub Copilot: flat price per user

Publicly sold as monthly plans per user
Light users subsidise heavy ones; average cost must stay well under the price

Fits predictable usage per user.

OpenAI's API: pay per token

Publicly priced per million input and output tokens
Cost follows usage exactly; customers forecast their own bill

Fits wildly varying usage.

What goes wrong

3 min

Estimating tokens from what users type

Prompts, tools, documents and history are 10–100× the message.

Instead

Log real usage per call; forecast from it.

Treating input and output as equal

Output often costs 4–5× more per token.

Instead

Price them separately; find the bigger side first.

Ignoring conversation growth

Cost per conversation grows faster than the number of turns.

Instead

Track p50/p95 per conversation; summarise history past a few turns.

Trade-offs

3 min

Option	Buys	Costs
Rich context every call	Best answers on hard questions; simple	Input cost and latency on every request
Lean, cached context	Cache discounts, predictable cost	Retrieval tuning; may drop a needed clause

Wednesday, 16:00. Planning the next cost round.

Maya · Product manager
I want the full policy pack and all history in every request. Fewer "I don't know"s.
Karim · CTO
That's +2,500 tokens a request, +7,500 € a month. Retrieve the relevant policy; summarise after 6 turns.
Maya · Product manager
And when retrieval misses the one clause a guest needed?

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The budget per unit you care about: AI cost per Plus member under a share of the 9 € plan.
The prompt layout: the 1,800-token stable prefix first and byte-identical; dates, names and ids after.
How long answers may be (output is ~59% after caching), and the cost-per-request alert.

Tell Claude Code

Build a cost forecast and guard for the Plus Assistant (~20,000 requests a day; 3,000 input with a 1,800 prefix, 400 output; 5/25/0.50 EUR per M). Plan first. One: `scripts/cost-forecast.ts` reads gateway usage logs and prints cost per request, month and member (5,000), split uncached, cached, output. Two: move everything that varies after the prefix in `src/concierge/prompt.ts`, with a test that the first 1,800 tokens match across requests. Three: an alert above 0.02 EUR per request.

Check before you accept

One line by hand: 0.025 € per request, ~15,000 € a month uncached.
After caching, output is the biggest slice; if not, the cache isn't hitting.
The prefix test compares two real requests with different guests and dates.

Red flags

The forecast multiplies the pilot invoice by 33.
A timestamp or request id added at the top of the system prompt.
Token counts guessed from text length instead of usage fields.

Check yourself

2 min

Q1/3Reasoning

A booking FAQ bot makes 10,000 requests/day with 2,000 input and 500 output tokens each, on a model priced at €1 per 1M input and €5 per 1M output tokens. No caching. What is the monthly cost (30 days) in euros?

€

How confident are you?

Next episode: traces show 70% of Plus Assistant questions are "what time is breakfast?", and Karim wonders why we pay the large model to answer them.

Finished?

Mark it complete to update your skill map.

Next: Routing for cost: small models for most, large models where it matters

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

From one request to a price per member

Worked example

Lina forecasts the rollout bill

Give Karim a defensible monthly number for the Plus Assistant at 20,000 requests a day, and say whether it fits the 9 € StayHub Plus plan (target: AI cost under 30% of the price per member, 5,000 members).

1Measure tokens, don't guess them
Pull a week of rollout-beta spans: median 3,000 input tokens (1,800 stable prefix) and 400 output tokens per request.
Why: Guests type 20 words; we send 3,000 tokens. Estimating from what users type is the classic mistake.
2Price input and output separately
Input: 3,000 × 5 € / 1M = 0.015 €. Output: 400 × 25 € / 1M = 0.010 €. 0.025 € per request.
Why: Output costs 5× more per token, so it's 40% of the money with 12% of the tokens.
3Apply caching
1,800 prefix tokens at 0.50 €/M = 0.0009 €; 1,200 uncached × 5 €/M = 0.0060 €; output 0.0100 €. 0.0169 € per request.
Why: The prefix is identical on every request, so caching cuts cost with no quality risk, as long as the prefix stays identical.
4Scale to a month
0.025 × 20,000 × 30 = 15,000 € without caching; 0.0169 × 600,000 = ~10,140 € with it.
Why: Showing both numbers tells the board what the caching work is worth (~4,860 €/month).
5Turn it into unit economics
Per member: 15,000 / 5,000 = 3.00 € (33% of 9 €) without caching; ~2.03 € (23%) with it.
Why: "10,140 €" doesn't say if the business works. "23% of the plan's price" does.
6Check the tail
Look at cost per conversation at p50, p95 and p99. The 5% of 25-turn conversations may be a large share of the bill, because history is re-sent every turn.
Why: Averages hide heavy users. The tail decides whether you need history summarisation or turn caps.

Turn	Input tokens	Output tokens	Cost (5 € / 25 € per M)
1	2,150	300	0.018 €
2	2,600	300	0.021 €
3	3,050	300	0.023 €
4	3,500	300	0.025 €
5	3,950	300	0.027 €
6	4,400	300	0.030 €
Total	19,650	1,800	≈ 0.143 €

A 2,000-token prefix, each turn adding 150 user + 300 assistant tokens. Total input grows roughly with the square of the number of turns.

Interactive simulation

Visual diagram

Guest

Assistant backend

Policy retrieval

Conversation store

AI gateway

LLM

Cost dashboard

Click a component

Where tokens come from in one request, and where the bill gets computed.

Analogy

Taking a taxiCalling an AI model

Taking a taxiBriefing the driver before every tripCalling an AI modelInput tokens: instructions, documents, history
Taking a taxiMinutes of driving, the pricey partCalling an AI modelOutput tokens the model writes
Taking a taxiLoyalty card for the same briefingCalling an AI modelPrompt caching for an identical start
Taking a taxiDriving back to the start at each stopCalling an AI modelRe-sending the whole chat every turn

Where the analogy breaks: a taxi charges the same for any words. An AI model charges per token, and some languages and formats use many more tokens for the same meaning.

Check your understanding

Q1/3Scenario

Your logs show: 1,200 input tokens per request (60% already cached) and 900 output tokens, on a model whose output price is 5× its input price. Which change most reduces cost?

How confident are you?

Practice ladder

Step 1 · Trace

Price the FAQ bot

Estimation Warm-up

StayHub's hotel FAQ bot (for hotel staff) makes 10,000 requests a day. Each sends 2,000 input tokens and gets 500 output tokens. The small model costs 1 € per 1M input and 5 € per 1M output tokens. No caching.

Trace the cost: per request, per day, per 30-day month. Which side (input or output) is bigger?

Step 2 · Complete

Finish the conversation-growth table

Estimation Practice

Each turn re-sends a 2,000-token prefix, all previous turns (each adds 150 user + 300 assistant tokens), and the new 150-token message. Output is 300 tokens per turn. Prices: 5 € per 1M input, 25 € per 1M output. No caching.

Turn	Input tokens	Cost
1	2,150	0.018 €
2	2,600	0.021 €
3	___	___
4	___	___
5	3,950	0.027 €

Fill in turns 3 and 4, then say roughly what a 10-turn conversation costs compared with 10 × the first turn.

Step 3 · Solve

Fix the prompt builder that defeats the cache

Code Practice

After a refactor, cached input dropped from ~60% to 0% and the bill rose. Here's the new prompt builder. Find everything that breaks the cache, and rewrite the order.

function buildPrompt(guest: Guest, hotel: Hotel, history: Msg[], msg: string) {
  return [
    `Guest: ${guest.name}, loyalty tier ${guest.tier}`,
    `Request id: ${crypto.randomUUID()}`,
    SYSTEM_INSTRUCTIONS,
    shuffle(TOOL_DEFINITIONS).join("\n"),
    hotel.policies,
    ...history.map(format),
    msg,
  ].join("\n\n");
}

Step 4 · Decide

What to do about 25-turn conversations?

Decision Stretch

Traces show 5% of Plus Assistant conversations reach about 25 turns (long stays, group trips). Using the growth model from the table (2,150 input on turn 1, +450 per turn, 300 output, 5 €/25 € per M, ignore caching):

A 4-turn conversation costs about 0.09 €.
A 25-turn conversation costs about 1.13 €.

Options: (A) do nothing; (B) cap conversations at 12 turns and start a new chat; (C) after 6 turns, replace older history with a ~600-token summary.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a product manager how you'd estimate the monthly cost of a new AI feature before launch, and what would make the estimate wrong.

0 words