Context engineering: deciding what the model gets to see
Thursday, 15:20: at turn 60 of a chat about two bookings, the concierge changed Ana's Porto booking when she meant Lisbon, and denied the €30 late checkout it had promised her on turn 12. On turn 1 it was flawless.
The model only knows what you show it on each call. Choosing what goes in, and what stays out, is a skill of its own.
- Problem
An assistant must answer well on turn 1 and on turn 60, with the right policy, the right booking, the right tools, at an acceptable cost and latency.
- Why the simple system fails
Stuffing everything in (all tools, all docs, full history) makes calls slow, expensive and less accurate; key facts get lost in the middle, old turns crowd out rules, and a pasted web page can hijack instructions.
- Engineering concept
Treat the window as a budget. For each item decide: pin it, retrieve it just in time, summarize it, or exclude it. Order deliberately, compact as you go, and isolate untrusted text.
- Technology
Context assembler with per-section token budgets, retrieval (RAG), running summaries, tool selection, prompt caching on a stable prefix, delimiters and privilege separation for untrusted content, sub-agents with fresh contexts.
- Trade-offs
Every exclusion risks missing something; every inclusion costs tokens, latency and attention. Summaries lose detail; retrieval can miss; caching rewards a stable prefix that resists per-turn tweaks.
The problem
2 min- Maya · Product managerTurn 60, Ana: "can you move it to the 18th? And the €30 late checkout still stands?" She meant Lisbon.
- Omar · On-call SREIt moved her Porto check-in, and denied the late checkout it agreed on turn 12.
- You · Engineer at StayHubBut every call fits the budget now.
The big idea
4 minGive the model the smallest set of high-signal tokens that makes the right answer likely, rebuilt on every call.
A receptionist going off shift doesn't hand over every word said at the desk. She leaves a note: "Ana Ribeiro, two bookings. Active: Lisbon, leaving 16th. Promised late checkout, €30." The next person reads that first.
How it works
5 mindecides, on every call, what goes into the window and in what form. For each piece of information, one of five choices:
- 1Pin it
Needed every turn: rules, format, and pinned state, a ~300-token block rebuilt from our data each turn: active booking, promises made, open tasks.
- 2Retrieve it
Needed on some turns: cancellation policy, past stays. Fetched just in time, filtered by hotel.
- 3Summarise it
: old turns become a structured summary that keeps ids, prices and promises verbatim. Ours said "discussing Lisbon and Porto" and lost the €30.
- 4Isolate it
Untrusted content, text a third party could have written (a pasted web page), is wrapped, capped, and never trusted with write tools.
- 5Leave it out
Everything else, with a tool to fetch it if it turns out to matter. Only the ≤ 6 tools for this stage, not all 30.
- 6Order it
Stable content first (system, tools) so prompt caching works. Key facts near the end: models use the start and end best ().
Watch a decision being made
4 minYou re-engineer the concierge's context
Make long chats as reliable as short ones, at about 7,000 tokens per call. You lead; Lina reviews at the end.
- 1Sort every item into a choice
Rules: pin. Active booking and commitments: state block. Policies: top-2, by hotel. Old turns: structured summary. Pasted pages: isolate. Tools: ≤ 6 for this stage.
Why: Treating everything as "the last N turns" is how the €30 promise got lost.
- 2Build the state block from data
Rebuilt every turn, ~300 tokens.
text<state> Active booking: LIS-2280, Lisbon, 12–16 July Other booking: POR-3315, Porto, 16–19 July Commitments: late checkout LIS-2280, €30 (agreed turn 12) </state>Why: It's correct even when the summary isn't.
- 3Order, then remind
System and tools, state, docs, summary, last 6 turns, a one-line reminder ("Active booking LIS-2280; changes need a price quote and confirmation"), the message.
Why: A byte-identical prefix keeps the cache; the reminder sits where attention is strongest, for ~100 tokens.
- 4Measure on long chats
Log tokens per section; run 50 tests of 40+ turns. 72% → 94% correct, ~7,000 tokens. Lina: "Checked. Ship it."
Why: Context bugs only show in long chats.
In real life
3 minChatGPT Memory: pin facts across chats
- Launched in 2024: remembers facts you share and brings them into later chats
- Pinned facts are present even when irrelevant, and can go stale
Fits small, stable facts users expect remembered.
Cursor: fetch relevant code just in time
- The AI code editor indexes your codebase and pulls in the files relevant to each request
- Retrieval can miss the one file that mattered
Fits large material where each request needs a slice.
What goes wrong
3 minIt removes the system prompt or the turn naming the booking.
Pin rules and state; compact old turns; keep raw turns in storage.
Refs, prices and promises vanish.
A structured summary, tested on long chats.
A timestamp in line one defeats prompt caching.
Stable content first, changing content later.
Trade-offs
3 min| Choice | Fits | You pay |
|---|---|---|
| Pin | Small facts needed every turn | Paid every call; can go stale |
| Retrieve | Large or changing knowledge | Can miss; latency and evals |
| Summarise | History past its budget | Lossy; errors persist |
| Isolate | Subtasks with lots of or untrusted text | More calls; harder to debug |
- Maya · Product managerPin their whole profile: 40 past stays, reviews, preferences. About 6,000 tokens.
- Karim · CTOThat doubles every call for every returning guest. Pin nothing about past stays.
- Maya · Product managerThen the concierge asks a guest with 40 stays if she prefers a quiet room. Again.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The token budget per section (about 7,000 total) and what's cut first.
- What a summary must never lose: booking refs, prices, dates, every promise.
- How outside text is fenced, and which tools stay available while it's in context.
Long concierge chats go wrong: at turn 60 Ana's "move it to the 18th" moved her Porto check-in instead of Lisbon, because the summary dropped both refs and the €30 promise. Rework the context assembler, not the model. Order: rules, ≤ 6 tools for this stage, a state block of active bookings and promises, top-2 policy chunks, a structured summary keeping refs, prices and promises verbatim, the last 6 turns, the message. Fence pasted text. Keep the prefix byte-identical. Plan first; report tokens per section and accuracy on the 50 long chats.
- The assembled context for turn 60 has both bookings and the €30 promise in the state block.
- The total is around 7,000 tokens, no section over budget.
- Nothing that changes per call sits before the stable prefix.
- It switches to a bigger window and sends everything.
- It truncates old turns with no summary.
- It keeps write tools available while pasted web text is in context.
Check yourself
2 minNext episode: a guest searches reviews for "quiet room to work remotely" and gets nothing, even though the perfect review says "peaceful, fast wifi, great desk".
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You re-engineer the concierge's context
Make long concierge chats as reliable as short ones: the right booking, promises kept, rules followed, safe with pasted text, at about 7,000 tokens per call. You lead; Lina reviews at the end.
- 1Sort every item into a choice
Rules and format: pin. Active booking and commitments: pin (state block). Policies: retrieve top-2, filtered by hotel. Old turns: structured summary. Pasted pages: isolate. 30 tools: only the ≤ 6 for this stage of the conversation.
Why: Each item has a different need. Treating them all the same ("send the last N turns") is how the €30 promise got lost.
- 2Build the state block from data
Rebuilt every turn, ~300 tokens.
text<state> Guest: Ana Ribeiro Active booking: LIS-2280, Lisbon, 12–16 July Other booking: POR-3315, Porto, 16–19 July Commitments: late checkout LIS-2280, €30 (agreed turn 12) Open task: confirm taxi time </state>Why: It comes from the booking system and a stored commitments list, so it's correct even when the summary isn't.
- 3Order for caching and attention
System and tools first (stable, cacheable). Then state, docs, summary, last 6 turns, a one-line reminder ("Active booking LIS-2280; changes need a price quote and explicit confirmation"), then the message.
Why: Prompt caching needs a byte-identical prefix. And models use the start and end of a long context best ().
- 4Compact with structure
When history passes its budget, a small model summarises older turns, keeping refs, prices, promises and open tasks verbatim. Raw turns stay in storage behind a lookup tool.
Why: Summaries are lossy. A structured prompt keeps what matters, and storage keeps the rest reachable.
- 5Isolate pasted text
Wrap pasted pages in
<untrusted_document>tags, strip hidden text, cap at 1,500 tokens, and block write tools (likeadd_booking_note) on any turn that includes untrusted content unless the guest confirms.Why: Third-party text can carry instructions (). Wrapping lowers the odds; gating the tool removes the damage.
- 6Measure it
Log tokens per section on every call, and run 50 long-conversation tests (40+ turns). Result: 72% → 94% correct, ~7,000 tokens per call. Lina: "Checked. Ship it."
Why: Context bugs show up only in long chats. Without a long-chat test set, you'd only find them in production.
| Section | Budget (tokens) | Policy | Position |
|---|---|---|---|
| System prompt | 1,500 | Pinned; changed only on deploy | First (stable prefix) |
| Tools for this stage | ≤ 6 × ~200 | Pinned per stage, never all 30 | After system (stable prefix) |
| State block | ~300 | Rebuilt every turn from our data | After the prefix |
| Policy chunks | 2 × ~500 | Retrieved just in time, filtered by hotel | Near the question |
| Structured summary | ≤ 400 | Compacted from older turns | Before recent turns |
| Last 6 turns | ~2,400 | Verbatim, sliding window | Before the message |
| Reminder | ~100 | This turn's key rule and state | Just before the message |
| Guest message | capped | Always; pasted pages wrapped | Last |
Interactive simulation
Visual diagram
A context assembler builds each call from budgeted sources, in a deliberate order, and keeps untrusted text fenced off.
Analogy
- Briefing a consultant for a meetingThe consultant forgets every past meetingContext engineeringThe model is stateless
- Briefing a consultant for a meetingOne briefing folder per meetingContext engineeringThe context window
- Briefing a consultant for a meetingA one-page standing briefContext engineeringPinned rules and identity
- Briefing a consultant for a meetingNotes on past meetings and open itemsContext engineeringA compact history summary
- Briefing a consultant for a meetingA phone number to ask for moreContext engineeringTools and retrieval
- Briefing a consultant for a meetingA sleeve marked "their words, not ours"Context engineeringFenced-off untrusted content
Where the analogy breaks: a consultant can tell a contract from a sales pitch. A model can't reliably tell whose words are whose, so labels and separation must be done for it.
Check your understanding
Practice ladder
Trace a turn through the assembled context
Scenario Warm-upAt turn 48, the concierge's context contains:
[system] … Changes to bookings need a price quote and explicit confirmation.
[tools] search_rooms, quote_change, change_booking, add_booking_note
<state> Active booking: LIS-2280 (12–16 July). Other: POR-3315 (16–19 July).
Commitments: late checkout LIS-2280, €30 (turn 12). </state>
[summary] … guest compared restaurants in Porto (turns 30–42) …
[last 6 turns] all about Porto restaurants
[reminder] Active booking LIS-2280. Quote and confirm before any change.
[guest] "Actually, can we make it one night longer?"Trace what the concierge should do, step by step, and which part of the context drives each step.
Finish the compaction prompt
Code PracticeThe compaction prompt that blurred the €30 promise said only: "Summarise the conversation briefly." Finish Lina's replacement: fill in the keep and drop lists.
Summarise the conversation so far for an assistant that will continue it. Keep, verbatim where possible: - ___ - ___ - ___ - ___ Drop: ___ Max 300 words.
Context for a group-booking assistant
Design StretchThe sales team gets an assistant for group bookings (weddings, conferences). One task can take 30+ tool calls: checking availability in 8 hotels, reading each hotel's group policy (5,000 tokens each), comparing quotes, drafting an offer. Each search_availability result is about 15,000 tokens of JSON. The client's request email sometimes includes attachments from third parties.
Design the context: what the main conversation sees, how tool outputs are handled, where the hotel policies go, and how third-party attachments are treated.
Raw tool output or a summary?
Decision StretchThe concierge's search_rooms tool returns all matching rooms: on average 12,000 tokens of JSON (photos, amenities, every rate plan). A teammate wants to pass it raw: "the model can pick what it needs, and we never lose information". The alternative: return the top 5 rooms with id, name, price, 3 key features and cancellation terms (~600 tokens), and keep the full result behind get_room_details(id).
Searches happen in ~30% of 60,000 daily conversations, and results stay in the history for later turns.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
A colleague says: "The new model has a huge context window, so we'll just send everything." Explain what you'd do instead and why.