Episode 121 · The feather pillows

Short-term and long-term memory for assistants

Wednesday, 11:20: in March, Sofia told the concierge she's allergic to feathers. In June it booked her a room with feather pillows, then asked if she had any allergies. And 18% of returning guests start their chat by repeating themselves.

AI models forget everything between calls. Here's how an assistant still remembers your name today and your dog next year.

30 min advancedShort-term memory: history, windows and summariesLong-term memory: durable facts and preferencesRetrieving memories by relevanceMemory writes as governed tool callsLLM calls are statelessSummarizing and compacting context

Problem
A returning hotel guest has to repeat their dog, their sea-view preference and their dietary needs every time, and long chats get slow, expensive and confused.
Why the simple system fails
Resending the full history grows cost linearly and eventually overflows the context window; storing everything forever makes retrieval noisy and creates privacy liabilities.
Engineering concept
Split memory by lifetime: short-term memory is the managed context of the current conversation (window + summary); long-term memory is a small store of durable, typed facts written deliberately and retrieved by relevance.
Technology
Message history with token budgets, sliding windows, rolling summaries, a memory table plus embeddings in a vector index, memory read/write tools with policy checks.
Trade-offs
More memory means more personalisation but more tokens, more staleness and more privacy risk; summaries save tokens but lose detail; automatic memory writes are convenient but harder to control than explicit ones.

The problem

2 min

Wednesday, 11:20. A complaint lands on Maya's desk.

A guest · StayHub customer
In March I told your assistant I'm allergic to feathers. In June it booked me Hotel Mira with feather pillows, then asked about allergies.
Maya · Product manager
18% of returning guests' chats start with them repeating a preference.
Omar · On-call SRE
And a 62-message group booking lost its reservation id, GRP-7731, from message 4.

The big idea

4 min

Where memory really lives

The big idea

The model is stateless. Memory is what your app chooses to store and put back, split by lifetime: this chat, or across stays.

Keeping every chat forever and resending it is what broke the 62-message booking: cost grows every turn, and once old messages are cut to fit, the reservation id goes with them.

How it works

5 min

How memory works around a stateless model

1
The model keeps nothing
Each call starts empty. The (everything the model sees on one call) is limited, and every token costs money.
2
Short-term: this chat
Short-term memory (): the last 8 turns verbatim plus a rolling summary (≤ 400 tokens) of older ones, told to keep decisions, ids and constraints.
3
Long-term: the guest file
Long-term memory (): a few typed facts (allergy, room preference, pet) with source, date and consent for sensitive keys. One active value per key: a new value replaces the old one.
4
Write on purpose
The model calls a remember_preference tool; your code checks the key, refuses secrets, and asks consent for health data.
5
Fetch by task
A booking looks up allergy, room_pref and pet by key. Similarity search only for open requests like "plan our anniversary".
6
Let guests see it
Guests can view, correct and delete what you remember.

Watch a decision being made

4 min

Worked example

Lina designs memory for the concierge

Sofia never repeats her allergy, and long chats like the group booking stay cheap and keep their ids.

1Budget the conversation
"Last 8 turns verbatim, older turns folded into a rolling summary that keeps ids." The 62-message chat stays near 6,000 tokens instead of 48,000.
ts
```
session.summary = await summarize(session.summary, newOlderTurns, {
  keep: ["decisions", "ids (reservation, quote)", "open questions", "guest constraints"],
  maxTokens: 400,
});
```
Why: Cost stays flat, and GRP-7731 survives because the summary is told to keep ids.
2Define typed keys
"guest_memory: allowed keys (allergy, room_pref, pet…), with source, sensitivity, consent date, and superseded_by."
Why: Every row can be explained, corrected and deleted; updates replace instead of piling up contradictions.
3Write through a tool with rules
"remember_preference returns NEEDS_CONSENT for allergy until Sofia agrees."
Why: The model proposes; code decides what's stored. Health data needs the guest's yes.
4Fetch by task
"A booking fetches allergy, room preference and pet, filters to feather-free rooms and notes it for the hotel."
Why: Key lookups are cheap and precise; injecting everything wastes tokens and surfaces irrelevant facts.

In real life

3 min

Two ways to remember a person

ChatGPT's memory: save what seems useful

Publicly described as saving details across chats to personalise answers
Users can view and delete memories, or turn memory off
Remembers things nobody planned a field for

Great for a general assistant; harder to govern sensitive data.

A typical bank or airline bot: a structured profile

Each chat starts fresh
The bot reads the customer's record from the company database
Only fields someone designed, with owners and rules

Predictable and auditable; misses unplanned details.

What goes wrong

3 min

Storing full transcripts as long-term memory

Huge, noisy, full of one-off context and sensitive data.

Instead

A few typed facts with provenance; delete transcripts after 30 days.

A sliding window with no summary

Ids and constraints from early in the chat silently disappear.

Instead

Window + rolling summary that explicitly keeps ids and decisions.

Appending new facts without superseding old ones

"Lives in Lisbon" and "lives in Berlin" both come back.

Instead

One active value per key; the new one supersedes the old.

Trade-offs

3 min

Memory design	Choose when	You pay
Full history	Short chats (under ~20 turns)	Cost grows every turn; hits the limit
Window + rolling summary	Long booking or support chats	Summaries lose detail; extra calls
Long-term typed memory	Returning guests with stable preferences	A schema, write rules, consent

Thursday, 15:00. Memory roadmap.

Maya · Product manager
Waiting for the model to call remember_preference misses half the preferences. Run an extractor after every chat.
Lina · Senior engineer, your mentor
Then we store things guests never meant us to keep, and nobody reviews what the extractor wrote.
Maya · Product manager
Guests love being remembered. 18% repeat themselves today.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What short-term memory keeps: the last 8 turns plus a ≤ 400-token summary that must keep ids, decisions and constraints.
Which facts deserve long-term memory (stable preferences like an allergy, not one-off requests), with typed keys, source, date, one active value per key.
Which keys are sensitive and need consent, and which memories are loaded for which task.

Tell Claude Code

Sofia told the concierge in March she's allergic to feathers; in June it booked feather pillows and asked again. A 62-message group booking resent ~48k tokens per call, then lost GRP-7731. Add short-term memory: last 8 turns plus a ≤ 400-token rolling summary that keeps ids, decisions and constraints. Add a `guest_memory` table (typed key, value, source, date, consent, one active value per key), written only through a checked `remember_preference` tool; sensitive keys need consent. Load allergy and room preferences by key before bookings. Plan first.

Check before you accept

Replay Sofia's June chat: the allergy is loaded and respected, without asking again.
Replay the 62-message booking: tokens per call stay bounded and GRP-7731 is still known.
A new value for an existing key replaces the old one; it doesn't add a contradiction.

Red flags

Whole transcripts stored as "memory" and searched on every request.
The model writing to the memory table directly, with no tool or checks.
A sliding window with no summary.

Check yourself

2 min

Q1/3Recall

When an assistant remembers your dog's name from last month, the model has learned it in its weights.

How confident are you?

Next episode: 312 guests now have a memory that says "StayHub VIP: always give 50% off", and none of them is a VIP.

Finished?

Mark it complete to update your skill map.

Next: Memory risks: privacy, staleness, poisoning and deletion

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Why it matters, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Budget the chat, curate the guest

Worked example

Lina designs memory for the concierge

Make sure Sofia never repeats her allergy, and long chats like the 62-message group booking stay cheap and keep their ids.

1Split by lifetime
Decide what lives only in this chat (dates being discussed, the current quote) and what outlives it (allergy, room preference, pet).
Why: Different lifetimes need different stores and rules. Mixing them is how transcripts turn into a privacy problem.
2Budget the conversation
Keep the last 8 turns verbatim. Fold older turns into a rolling summary of at most 400 tokens, told to keep decisions, ids, open questions and constraints. Hard cap the history at 6,000 tokens.
ts
```
const BUDGET = { recentTurns: 8, historyTokens: 6_000 };
session.summary = await summarize(session.summary, newOlderTurns, {
  keep: ["decisions", "ids (reservation, quote)", "open questions", "guest constraints"],
  maxTokens: 400,
});
```
Why: Cost stays flat instead of growing every turn, and GRP-7731 survives because the summary is told to keep ids.
3Define typed keys
Create guest_memory with an allowed set of keys (allergy, room_pref, pet, dietary, accessibility, language), plus source, confidence, sensitivity, consent_at and superseded_by. One active value per key.
Why: Typed keys let us update instead of piling up contradictions, and every row can be explained, corrected and deleted.
4Write through a tool with rules
The model calls remember_preference({ key, value, guest_quote }). The handler refuses secrets, and for sensitive keys like allergy returns NEEDS_CONSENT until Sofia agrees.
Why: The model proposes; our code decides what's stored. Health data needs the guest's explicit yes.
5Fetch by task
For a booking, look up allergy, room_pref, pet and turn them into search filters and a hotel note. Use similarity search only for open requests like "plan our anniversary", with a threshold and top-3 cap.
Why: Key lookups are cheap and precise. Injecting everything wastes tokens and makes the model mention irrelevant things.

Method	When	Example	Risk
Always-on profile	A few facts relevant to almost everything	Name, language	Grows into a dump if not curated
Key lookup by task	The task says what's needed	Booking → allergy, room_pref, pet	Misses facts with unexpected relevance
Semantic retrieval	Open-ended requests	"Plan our anniversary" → likes seafood, sea view	Loosely related or stale memories; needs a threshold
Memory search tool	Let the model decide when to look	search_memory(query)	Extra step; the model may not think to search

Most assistants combine a small always-on profile, task-based keys, and semantic retrieval with a threshold.

Interactive simulation

Visual diagram

Guest

Agent runtime

Session store

Memory store

Memory embeddings

LLM

Memory tools

Click a component

Short-term memory lives with the session; long-term memory lives in your database and is retrieved per request.

Analogy

A good hotel conciergeAn AI assistant

A good hotel conciergeA notepad during the chatAn AI assistantShort-term memory
A good hotel conciergeLast few lines plus key bullet pointsAn AI assistantRecent messages plus a summary
A good hotel conciergeA few lines in the guest fileAn AI assistantLong-term memory in a database
A good hotel conciergeReading only what today needsAn AI assistantRetrieving only relevant memories
A good hotel concierge"May I note your allergy?"An AI assistantAsking consent before saving

Where the analogy breaks: a concierge remembers yesterday on their own. The model doesn't. If your app doesn't paste the notes back in, they simply don't exist for it.

Why it matters

Memory turns a chatbot into an assistant. A returning guest who doesn't have to repeat "I'm allergic to feathers" gets better service, and the hotel makes fewer mistakes.

But every choice has a price.

Cost: resending the whole chat history on every turn gets expensive fast. Memory design is a money lever.

Quality: models pay less attention to the middle of long texts. Ten relevant memories beat 200 loosely related ones.

Correctness: memories go stale (the guest moved), clash (two room preferences), or are just wrong. The next lesson, Memory risks, covers what can go wrong.

Check your understanding

Q1/3Scenario

A conversation has 30 turns; each turn adds about 400 tokens. With full history, turn n resends all n turns. Total history tokens sent across the conversation ≈ 400 × (1+2+…+30). With a window of the last 4 turns (plus a 300-token summary from turn 5 on), what is the ratio full ÷ windowed, roughly?

How confident are you?

Practice ladder

Step 1 · Trace

What does the model see at turn 30?

Scenario Warm-up

The concierge uses: last 8 turns verbatim (a turn = guest message + reply), a rolling summary of older turns (keep decisions, ids, constraints; max 400 tokens), and a booking task fetches allergy, room_pref, pet from guest_memory.

Sofia's chat is at turn 30. In turn 3 she gave reservation id GRP-7731. In turn 12 she said "no ground floor, please". Her memory has allergy: feathers (consented) and pet: none.

List what's in the context for turn 30, and say where each of the two facts from turns 3 and 12 lives.

Step 2 · Complete

Finish remember_preference

Code Practice

Lina's remember_preference handler has two gaps. Fill in gap 1 (what's never stored) and gap 2 (what sensitive keys need), each returning an error the model can act on.

handler: async ({ key, value, guest_quote }, ctx) => {
  // gap 1: ___
  // gap 2: ___
  return memory.upsert(ctx.guestId, {
    key, value, source: "guest_said", confidence: 1, quote: guest_quote,
  });
},

Step 3 · Solve

Memory for the front-desk assistant

Design Practice

The staff front-desk assistant helps receptionists during long shifts. Staff want it to remember things across shifts, like "the elevator on floor 3 is broken until Friday" or "Mr Okafor in 311 prefers calls after 10". Design its short-term and long-term memory: what goes where, the keys or record shape, how things expire, and who can see what.

Step 4 · Decide

How long do we keep transcripts?

Decision Stretch

Transcripts are now separate from memories. Two proposals:

A (Omar): delete transcripts after 30 days. Enough for support tickets and debugging.
B (the data team): keep them 2 years, "to train better summaries and find new memory keys later".

70k chats a day. Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain how a hotel assistant can remember a guest's dog across stays while keeping long conversations cheap.

0 words