Episode 99 · Who is "it"?

Transformers, intuitively

Wednesday, 8:40: the old support classifier sent 1,200 guest messages to the wrong team last week, and the long ones fail 14 times more often. A complaint about a too-small suite went to the "anniversary gifts" team.

Follow a sentence through the engine inside every modern AI model, and see the one trick, attention, that made it all work.

30 min intermediateSelf-attention: tokens looking at other tokensLLMs predict the next tokenPretraining, instruction tuning and RLHFFoundation models: one pretrained model, many tasksTraining vs inference: learning weights vs using them

Problem
To continue "The guest booked a suite, but it was too small, so she asked to change it", a model must know that "it" means the suite, 12 words back.
Why the simple system fails
Older sequence models (RNNs) read one word at a time, squeezing everything into a single running memory: distant details fade, and training can't be parallelised across the sentence.
Engineering concept
Attention: every token directly looks at every earlier token and pulls in what's relevant, all positions computed in parallel. Stack many such layers.
Technology
Transformer models: embeddings, multi-head self-attention, feed-forward layers, trained on GPUs at massive scale, then aligned with instruction tuning and RLHF.
Trade-offs
Attention cost grows with context length, models are huge to train and serve, and their behaviour comes from data and training, not from rules you can inspect.

The problem

2 min

Wednesday, 8:40. The support team's weekly review.

Omar · On-call SRE
The old classifier misrouted 1,200 of 40,000 messages last week. Under 30 words: 1% wrong. Over 80 words: 14%.
Maya · Product manager
This one waited 6 hours with the gifts team: "I booked a suite for my parents' 40th anniversary in Porto… but it was too small, so can you move them?"

The big idea

4 min

How "it" finds the suite

The big idea

Attention: every token directly reads every earlier token, so distance stops mattering, and everything in the context can steer the answer.

The old model plays whispers: each person hears only the one before. A transformer is a meeting where anyone can look straight at whoever spoke earlier and ask "what did you mean?".

How it works

5 min

How a transformer reads the suite complaint

1
Tokens
The turns the text into ids: [I, booked, a, suite, …, it, was, too, small].
2
Embeddings
Each id becomes a list of a few thousand numbers, a , plus its so order matters.
3
Attention
: "it" looks back at every earlier token and pulls in a weighted blend. "suite" gets about 71% of its weight.
4
Many layers
A layer is attention plus a small network. Dozens are stacked: grammar first, then meaning ("the guest is unhappy with the room").
5
Next-token scores
The last token's vector scores every possible next token. All positions are computed at once on a GPU, which is why could train on vast text.
6
From text continuer to assistant
on huge text makes a text continuer; makes it answer requests; preference tuning () shapes tone and safety.

Watch a decision being made

4 min

Worked example

You estimate the cost of reading whole threads

The new router wants to send whole email threads (up to 8,000 tokens) instead of the latest message (~1,000). Estimate what that does to speed and cost; Lina checks.

1Count the attention pairs
Attention scores every pair of tokens: 1,000² = 1M vs 8,000² = 64M, 64× the attention work.
python
```
for n in (1_000, 4_000, 8_000):
    print(n, n * n / 1e6, "M pairs")   # 1, 16, 64
```
Why: Attention is quadratic in context length: the price of letting every token see every other.
2Check what the provider charges
Input is billed per token: 8× the tokens, 8× the input cost. Time to first token grows too.
Why: Tricks like the absorb some work, but you still pay per token and wait longer.
3Ask what the task needs
Routing needs the latest message, maybe the previous one, not the hotel's internal notes from three weeks ago.
Why: Everything in context can steer the answer. Less, well-chosen text is cheaper and safer.
4Decide, then measure
Send the latest message plus a short thread summary (~1,500 tokens). Measure routing accuracy with and without the summary before keeping it.
Why: A summary keeps useful context without paying for, or being steered by, everything else.

In real life

3 min

How much of the stack to build?

Bloomberg: pretrained its own model

Publicly described BloombergGPT (2023), trained on decades of its financial data plus public text
Months of work, a lot of compute, a dedicated team

Fits massive specialised data and the means to train.

Notion: built on foundation models

Notion AI is publicly known to run on models from outside AI labs
Adapts them with prompts and the user's own pages; can switch models

Fits most products: the lab buys the capability, you add context.

What goes wrong

3 min

Thinking attention means careful, human reading

Anything in context, including forwarded or untrusted text, can steer the output.

Instead

Send only what the task needs; strip internal notes; mark untrusted text as data.

Assuming a big context is free

More tokens: more delay, memory and cost per call.

Instead

Measure time to first token as context grows.

Fine-tuning to teach the model your hotels' facts

Unreliable for changing facts; redone whenever they change.

Instead

Facts in the context; for format, tone or cost on a stable task.

Trade-offs

3 min

Option	Fits when	You pay
Foundation model + prompts	Almost always first	General-model price per call; vendor sets behaviour
Fine-tune on your data	Stable, high-volume, narrow task	Thousands of examples; retrain when requirements change
Pretrain your own	Rarely: massive specialised data	Enormous compute and months of work

Friday, 10:00. Replacing the 2021 classifier.

Karim · CTO
We have 2 million labelled tickets. Fine-tune a small transformer: cheap per call at 40,000 messages a week.
Maya · Product manager
Our categories change every quarter. Prompt a foundation model and ship next week.
Karim · CTO
Prompting a big model for every ticket costs more forever.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What goes into the context: the message and the categories, not the guest's whole history.
Where facts live: per request in the prompt, not in the weights by .
That the model reads everything you send, including instructions hidden in a guest's message (): routing output must be one of your categories.

Tell Claude Code

Our router misroutes 14% of messages over 80 words. Build an evaluation first: 2,000 labelled past tickets used for nothing else, real short/long proportions, at least 300 over 80 words. Write tools/routing-eval reporting accuracy for short and long messages separately, plus the worst mix-ups. Run it on the current classifier for the baseline. Then add a router calling our hosted LLM with the message and category list, answering exactly one category name. No guest names or emails in the prompt.

Check before you accept

The baseline reproduces ~1% short and ~14% long misroutes; otherwise the eval set isn't representative.
Long messages are reported separately.
Anything that isn't an exact category name falls back to a human queue.

Red flags

It sends each guest's full booking history "so the model understands better".
It uses the same tickets to tune the prompt and report the score.
It accepts free-text answers as routing categories.

Check yourself

2 min

Q1/3Reasoning

Which property most explains why transformers, rather than earlier recurrent networks, could be trained on vastly more data?

How confident are you?

Next episode: a 70-turn chat with a family planning a trip to Lisbon crashes with "context length exceeded", and Japanese guests' chats cost almost twice as much as English ones.

Finished?

Mark it complete to update your skill map.

Next: Tokens and context windows

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Attention, and what it costs

Worked example

You estimate the cost of reading whole threads

The new router wants to send whole email threads (up to 8,000 tokens) to the model instead of the latest message (about 1,000 tokens). You estimate what that does to speed and cost; Lina checks your reasoning.

1Count the tokens
Latest message: ~1,000 tokens. Whole thread: up to 8,000. The model reads all of them on every call.
Why: Every cost in a transformer starts from how many tokens are in the context.
2Count the attention pairs
Attention scores every pair of tokens: 1,000² = 1M pairs vs 8,000² = 64M pairs, 64× more attention work for the full read.
python
```
for n in (1_000, 4_000, 8_000):
    print(n, n * n / 1e6, "M pairs")   # 1, 16, 64
```
Why: Attention is quadratic in context length. That's the transformer's price for letting every token see every other.
3Check what the provider charges
Input is billed per token: 8× the tokens is 8× the input cost per call. The time to the first token also grows with the prompt.
Why: Providers absorb part of the quadratic work with tricks like the , but you still pay for every token and wait longer.
4Ask what the task needs
Routing needs the guest's latest message plus maybe the previous one, not the hotel's internal notes from three weeks ago.
Why: Everything in context can steer the answer. Less, well-chosen text is cheaper and safer.
5Decide
Send the latest message and a short summary of the thread (~1,500 tokens). Lina checks: "Good. Measure routing accuracy with and without the summary before you keep it."
Why: A summary keeps the useful context without paying for, or being steered by, everything else.

Stage	Data	What it adds	Who does it
Pretraining	Trillions of tokens of text and code	Language, world knowledge; a text continuer	Model labs (enormous compute)
Instruction tuning	Many curated conversations	Answers instead of continuing; follows roles and formats	Model labs; you, for narrow fine-tunes
Preference tuning (RLHF etc.)	Rankings of answer pairs	Helpfulness, safety, refusals, tone	Model labs
Your adaptation	Prompts, your documents, sometimes a small fine-tune	Your domain, your facts, your format	You, mostly at inference time

Each stage starts from the previous stage's weights. Order-of-magnitude, illustrative.

Interactive simulation

Visual diagram

Input text

Tokenizer

Embedding table

Self-attention

Feed-forward

Output layer

Sampler

Click a component

One forward pass of a decoder-only transformer, producing one next token. Generation repeats this loop.

Analogy

A meeting roomA transformer

A meeting roomEach word is a person in the roomA transformerEach token gets its own vector
A meeting roomOld way: whispering down a lineA transformerOlder chain-style networks (RNNs)
A meeting roomEveryone can hear everyone at onceA transformerSelf-attention
A meeting roomListening most to the relevant peopleA transformerAttention weights
A meeting roomThinking quietly after each roundA transformerThe feed-forward step in each layer
A meeting roomMany rounds of discussionA transformerDozens of stacked layers

Where the analogy breaks: in a real meeting, people can listen to anyone. In a text-generating model, each word may only listen to the words before it, never to words that come later.

Check your understanding

Q1/3Reasoning

Order one forward pass of a transformer generating a token:

1.Sample one token and append it
2.Score every vocabulary entry from the last token's vector
3.Feed-forward processing, repeated over many layers
4.Tokenize the text into ids
5.Attention: each token mixes in information from earlier tokens
6.Look up embeddings and add position information

How confident are you?

Practice ladder

Step 1 · Trace

Which "it" gets cancelled?

Scenario Warm-up

A guest writes: "I booked the family room and the parking spot. Can I cancel it but keep the room?"

One attention head's weights from the token "it":

family	room	parking	spot	cancel	keep
0.08	0.41	0.17	0.21	0.06	0.07

The concierge is allowed to cancel items directly. Trace what is likely to happen, and explain why from how attention works.

Step 2 · Complete

Put the transformer back in order

Explain Practice

Lina's explanation for the support team has the stages shuffled and one missing. Put them in order and fill in the gap.

Every possible next token gets a score; one is picked and appended.
Each id becomes a vector of numbers, with its position mixed in.
___ (missing)
The text is cut into tokens and turned into ids.
A small network refines each token's vector; this repeats for dozens of layers.

Step 3 · Solve

A guest pastes a 40-page PDF

Estimation Practice

Maya wants guests to paste documents into the concierge, like a 40-page conference schedule (~24,000 tokens), and ask "which day is free for a spa visit?". Today a typical concierge call has ~3,000 tokens of context.

Estimate how the bigger context changes the input cost per call and the attention work, say what else can go wrong, and propose a better design.

Step 4 · Decide

StayHub's tone in six languages

Decision Stretch

Maya wants the concierge to sound like StayHub (warm, brief, no exclamation marks) in six languages. Today the style rules take 900 tokens of the prompt, and the model still drifts in Japanese and Polish. Options:

A. Keep improving the prompt and add 3 example replies per language.
B. Fine-tune a model on 20,000 replies approved by the support team.
C. Pretrain a StayHub model.

Volume: 60,000 conversations a day. The style hasn't changed in two years. Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a teammate, without equations, how a transformer figures out the next word, and why a pretrained model isn't automatically a good chatbot.

0 words