Episode 100 · Context length exceeded

Tokens and context windows

Saturday, 21:30: a 70-turn chat with the Okafor family planning a week in Lisbon fails with "context length exceeded". The same week, Japanese guests' chats cost 1.8× as much as English ones.

Tokens decide what the model sees, what fits, what you pay and how long you wait. Here's how to budget them like a pro.

30 min intermediateTokens & tokenizationTokenization quirks: languages, numbers, code and IDsContext windowLong context: cost, latency and lost-in-the-middleCost & latency scale with tokensContext engineering

Problem
A hotel assistant must answer with the right policy, remember the conversation and call booking tools, all inside a fixed context window, at a cost per conversation finance can live with.
Why the simple system fails
The naive build stuffs everything in: full history, every tool, twenty documents. Calls get slow and expensive, hit the limit on long chats, and the model misses the one paragraph that mattered.
Engineering concept
Treat the context window as a budget measured in tokens. Count precisely, reserve room for the answer, and spend tokens on what the task needs.
Technology
Tokenizers (BPE), token-count endpoints, budget-aware prompt assembly, history summarisation, retrieval, prompt caching.
Trade-offs
More context can mean better grounding, but also higher cost, slower first token, and diluted attention. Less context risks missing the key fact.

The problem

2 min

Saturday, 21:30. The Okafor family has been planning a week in Lisbon for 70 turns.

Omar · On-call SRE
Turn 70: "context length exceeded". Every retry fails the same way.
Maya · Product manager
They were about to book three rooms. And Japanese chats cost 1.8× English ones. Same questions.
You · Engineer at StayHub
Can't we just switch to a model with a bigger window?

The big idea

4 min

What fills the window

The big idea

A context window is a budget, not a bucket: decide what each call needs, and give every part its share.

It's a suitcase with a weight limit, and the airline charges per kilo. Rules, tools, history and space for the answer all share it. Pack the whole wardrobe and you pay on every flight, until the case won't close.

How it works

5 min

What one concierge call puts in the window

1
Tokens aren't words
Most use : a vocabulary of frequent chunks learned mostly from English. Common words are one token; UUIDs and many languages split into many. Japanese: ~1.8× the tokens.
2
Fixed parts: 6,000
System prompt 1,500, 30 tool definitions 3,000, Lisbon policies 1,500. Sent on every call, used or not.
3
History: +400 per turn
Every earlier message and reply, resent because calls are stateless. The only part that grows without limit.
4
Answer reserve: 1,000
The holds input and output. Budget = 32,000 − 1,000 = 31,000 for input.
5
Count
Input at turn n = 6,100 + 400 × (n − 1). Turn 63: 30,900, fits. Turn 64: 31,300, overflow. Every turn before paid for the whole history again.
6
Budget the parts
A token budget splits the window between rules, tools, documents, history and answer. A summary rewrites older turns into far fewer tokens.

Watch a decision being made

4 min

Worked example

You put the concierge on a token budget

Never overflow the 32,000-token window, cut cost per call, and don't forget what guests said early in long chats. Lina checks the result.

1Reserve the answer, pin what matters
max_tokens 1,000. System prompt and the new message always go in; key facts (booking reference, dates, allergies) sit in a small pinned block.
Why: Output shares the window. Pinned means never trimmed: these are the facts whose loss causes harm.
2Shrink the fixed parts
Only the tools for the guest's intent (~4 tools, 400 tokens instead of 3,000) and the top-3 policy chunks (600 instead of 1,500).
Why: Fixed parts are paid on every call. They're the cheapest tokens to cut.
3Cap the history
The last 6 turns (2,400) plus a running summary of older turns (≤ 400).
text
```
Summary (turns 1–57): Okafor family, 2 adults + 3 kids, Lisbon
12–19 July, €250/night, one child has a peanut allergy, wants a pool.
```
Why: A summary keeps the history's facts at a fixed size.
4Count and check
1,500 + 400 + 600 + 400 + 2,400 + 100 ≈ 5,400 tokens at any turn, versus 31,300 at turn 64. Count with the model's tokenizer; log per language. Lina: "Now run the long-chat tests, especially the allergy."
Why: A budget you don't measure drifts, and a summary that drops the allergy is worse than an error.

In real life

3 min

Send everything, or pick a slice?

NotebookLM: read the whole sources

Answers from documents you upload; publicly said to run on a long-context Gemini model
Few users per document, high value per question

Fits low-volume, high-value questions over a few documents.

GitHub Copilot: a few relevant snippets

GitHub has publicly described choosing relevant snippets to fit a limited prompt
Huge volume; fast and cheap per call, but selection can miss context

Fits high-volume calls that each need a small slice.

What goes wrong

3 min

Forgetting output shares the window

Input + max output over the window: the call fails or the answer is cut.

Instead

Budget = window − max output.

Assuming 4 characters per token everywhere

Many languages tokenize less efficiently.

Instead

Measure tokens per language with the real tokenizer.

Using a huge window instead of choosing

Every call pays, and the middle gets less attention ().

Instead

Send what the task needs.

Trade-offs

3 min

Option	Fits when	You pay
Lean context (retrieve, summarise, pin)	High-volume chat, small slice per question	Summaries lose detail; more parts to test
Long context (send everything)	Low-volume, high-value tasks over one big document	Cost and delay every call; the middle gets less attention

Tuesday, 11:00. Picking the long-term fix.

Maya · Product manager
Switch to a model with a 1M-token window. Send everything. It never forgets.
Karim · CTO
At 60,000 conversations a day, that's paying for every old turn on every call.
Maya · Product manager
A summary will drop something. I'd rather pay than forget an allergy.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The token budget per call: 31,000 input, 1,000 for the answer, and a normal-call target around 5,400.
What gets cut first when a call would overflow: old turns into the summary, then low-ranked chunks. Never the system prompt or the current message.
How you count: with the model's own , per language.

Tell Claude Code

The concierge fails with "context length exceeded" from about turn 64 because we resend the full history and all 30 tools every call. Plan first. Budget: 31,000 input tokens, 1,000 for the answer. Normal call: system prompt 1,500, only the intent's tools (~400), 3 policy chunks (600), a summary of older turns (400), the last 6 turns (2,400), the new message. Count with the provider's token counting. If over budget, cut oldest turns into the summary, then lowest-ranked chunks; never the system prompt. Log tokens per part.

Check before you accept

Replaying the Okafors' 70-turn chat finishes, with input around 5,400 tokens.
The summary after 20 turns keeps names, dates and the booking.
One Japanese conversation counted next to an English one, with the real tokenizer.

Red flags

It switches to a 1M-token model and changes nothing else.
It estimates tokens as characters ÷ 4 for every language.
It truncates the prompt from the start, cutting the system prompt.

Check yourself

2 min

Q1/3Debugging

A model has a 128k-token window. Your prompt is 121k tokens and you set max_tokens to 16k. What happens?

How confident are you?

Next episode: since a config change, the date extractor misreads one email in fifty and every welcome message opens with the same sentence. Same knob, turned the wrong way twice.

Finished?

Mark it complete to update your skill map.

Next: Sampling, temperature, top-p and determinism

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Count tokens, then budget them

Count, don't guess. The 4-characters rule is fine for napkin maths. Production budgets use the model's own tokenizer or the provider's token-count endpoint, because tokenizers differ between model families.

Worked example

You put the concierge on a token budget

Make the concierge never overflow its 32,000-token window, and cut cost per call, without forgetting what guests said early in long chats. Lina only checks your result at the end.

1Reserve the answer
Set max_tokens to 1,000 and budget the input to 31,000. In practice, aim much lower.
Why: Output shares the window. Forgetting it is the classic overflow.
2Pin what must always be there
System prompt (1,500) and the new message (~100) are required. Key facts (booking reference, dates, allergies) go in a small pinned block.
Why: These are the things whose loss causes real harm. Pinned means they're never trimmed.
3Shrink the fixed parts
Send only the tools for the guest's intent (~4 tools, 400 tokens instead of 3,000) and the top-3 policy chunks (600 instead of 1,500).
Why: Fixed parts are paid on every call, even when unused. They're the cheapest tokens to cut.
4Cap the history
Keep the last 6 turns (2,400) plus a running summary of older turns (≤ 400), updated every 6 turns by a cheap model.
text
```
Summary (turns 1–57): Okafor family, 2 adults + 3 kids,
Lisbon 12–19 July, budget €250/night, one child has a
peanut allergy, wants a pool, prefers Alfama or Chiado.
```
Why: History is the only part that grows without limit. A summary keeps its facts at a fixed size.
5Count and check
1,500 + 400 + 600 + 400 + 2,400 + 100 ≈ 5,400 tokens at any turn, versus 31,300 at turn 64 before. Log tokens per call by language. Lina: "Now run the long-chat tests, especially the allergy."
Why: A budget you don't measure drifts. And a summary that drops the allergy is worse than an error message.

Input	Approx. tokens	Why it matters
1,000 English words	~1,300	The baseline rule of thumb (≈ 4 characters per token)
Same text in many other languages	~1.5–3× more	Higher cost and a faster-filling window in those markets
A UUID (36 characters)	~20–25	Use short references ("booking B1") in prompts
Pretty-printed vs compact JSON	Often 1.3–2× more when pretty	Spaces and newlines are tokens too
30 tool definitions	~3,000	Paid on every call, even if no tool is used
A 40-page policy PDF	~20–30k	Fits big windows, but slow, costly and distracting every turn

Order-of-magnitude, illustrative. Tokenizers differ between model families; measure with yours.

Interactive simulation

Visual diagram

Guest

Context assembler

System prompt

Tool definitions

Retrieved docs

Chat history

Token counter

LLM

Click a component

Context assembly for one call. Every source competes for the same token budget, and the answer needs room too.

Analogy

Packing a suitcase for a flightFilling a context window

Packing a suitcase for a flightThe airline's fixed bag sizeFilling a context windowThe context window
Packing a suitcase for a flightPaying per kiloFilling a context windowPaying per token
Packing a suitcase for a flightPassport and tickets, always packedFilling a context windowSystem prompt and the user's question
Packing a suitcase for a flightSpace left for souvenirsFilling a context windowRoom for the model's answer
Packing a suitcase for a flightVacuum-packing bulky clothesFilling a context windowSummarising old chat history
Packing a suitcase for a flightCan't find the charger in a full bagFilling a context windowMissing the key fact in clutter

Where the analogy breaks: a suitcase weighs the same for every traveller. Token counts depend on language and format, so the same sentence can cost very different amounts.

Check your understanding

Q1/3Scenario

Your system prompt plus tool definitions is 6,000 tokens and is sent on 2 million calls per month. At an illustrative $3 per million input tokens (no caching), what does that fixed prefix alone cost per month, in dollars?

How confident are you?

Practice ladder

Step 1 · Trace

When does this chat overflow?

Estimation Warm-up

The Porto concierge uses a model with a 16,000-token window and max_tokens of 800. Each call sends:

System prompt: 1,200 · tools: 2,400 · policies: 1,000
History: everything so far, ~350 tokens per earlier turn
New message: ~100

Trace the input at turns 1, 10 and 30, and find the first turn that overflows.

Step 2 · Complete

Finish the priority list

Design Practice

Lina's context assembler adds parts in priority order and drops optional ones that don't fit. Two entries are missing. Fill them in and say for each whether it's required or optional.

System prompt (required)
___
Pinned guest facts: booking reference, dates, allergies (required)
Top-3 policy chunks for this question (optional)
___
Last 6 turns (optional)
Tool definitions for this intent (optional)

Step 3 · Solve

Put the reviews assistant on a token diet

Estimation Practice

StayHub's hotel-manager assistant (helps owners reply to reviews) sends per call: system prompt 1,200, 20 tools × 300, top-8 review chunks × 700, full history averaging 5,000, new message 50; output ~200. It makes 1.5M calls a month. Illustrative prices: €3 per 1M input tokens, €15 per 1M output.

Estimate the monthly cost, propose changes that cut input by at least half without hurting answers, and estimate the new cost.

Step 4 · Decide

A 120-page group contract

Decision Stretch

The sales team negotiates group bookings (weddings, conferences) with a 120-page contract (~80,000 tokens). They want an assistant to answer questions like "what's our penalty if the group cancels 30 days out?". About 40 questions a day, each answer reviewed by a salesperson. A long-context model handles 200k tokens; prices are about 2× our concierge model.

Options: A. Send the whole contract each time to the long-context model. B. Chunk it and retrieve the top-3 passages for each question.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

A colleague says: "The new model has a 1M-token window, so we can drop retrieval and just send all our docs." How do you respond?

0 words