What is an LLM, really?
Monday, 10:05: in 40 test chats, StayHub's new concierge invented 3 hotel facilities, including a rooftop pool at Hotel Alfama in Lisbon. Hotel Alfama has no pool, and Maya wants to know where the model got it.
No magic inside: a language model guesses the next word-piece, over and over. Once you see that, cost, speed and made-up answers all make sense.
- Problem
A hotel wants an assistant that answers guests in natural language, at any hour, in any language.
- Why the simple system fails
Hand-written rules and keyword bots break on every phrasing nobody anticipated, and can't hold a conversation.
- Engineering concept
Train a huge network to predict the next token of text; generating one token at a time produces fluent answers.
- Technology
Transformer-based LLMs served behind an HTTP API: tokenizer, GPU inference, streaming responses.
- Trade-offs
Fluent is not the same as correct. Every call costs tokens and time, and the model remembers nothing you don't resend.
The problem
2 min- Maya · Product manager40 test chats. 3 times it invented a facility. It told a guest Hotel Alfama has a rooftop pool. It doesn't.
- You · Engineer at StayHubMaybe our hotel data for Alfama is wrong?
- Lina · Senior engineer, your mentorWe didn't give it any hotel data. Guess before we look inside.
The big idea
4 minAn LLM predicts plausible text, one token at a time; truth has to come from what you give it.
Picture a brilliant improviser with amnesia: it has read most of the internet, remembers nothing between calls, and if you ask about your hotel's pool without telling it, it improvises something that sounds right.
How it works
5 min"AI" is nested: learns patterns from examples, does it with large neural networks, and an is a deep network trained on huge amounts of text to predict the next piece of text.
- 1Text in
Our app sends the system prompt and the question. Calls are : the model sees only what this call sends, so the app must resend earlier turns.
- 2Cut into tokens
A splits text into , about 4 characters of English each: "Alfama" may be two, "pool" one.
- 3Score every next token
The model gives a score to each of tens of thousands of possible next tokens.
- 4Pick one
Usually a likely one. sets how adventurous the pick is. It adds no knowledge.
- 5Append and repeat
The token is added and the model runs again, about 75 tokens a second after ~0.5 s to the first. A 600-token answer takes ~8.5 s.
No step checks truth. With no facts about Alfama in the input, the model writes what travel text usually says, and many hotel descriptions mention pools. That's a : fluent, confident, wrong.
Watch a decision being made
4 minLina debugs Monday's three bugs
The 40 test chats showed an invented pool, a forgotten guest name, and 8-second answers. Explain each from how an LLM works, and fix it.
- 1Read what the model actually saw
Log the exact request. For the pool: a one-sentence system prompt and the question. No hotel facts.
Why: The model can only use what's in the call. Reading the payload ends the guessing.
- 2Ground the facts
Add Alfama's real facts and a rule for gaps.
textAnswer only from the facts below. If they don't say, reply "I'm not sure, let me check with the hotel." Facts: breakfast 7:00–10:30; no pool; parking €18/night.Why: With the true fact in the input, the most plausible continuation is the true one.
- 3Resend history, shorten answers
The app stores each chat and resends earlier turns. Ask for 2–3 sentences, cap at 150, stream.
Why: The model "remembers" the guest's name only if it's in this payload. An 80-token answer takes ~1.6 s, and streaming shows the first word after 0.5 s.
- 4Rerun the 40 chats
0 invented facilities, names kept, median answer 1.8 s.
Why: A fix you haven't rerun is a hope. The 40 chats become the first test set.
In real life
3 minChatGPT at launch (2022): from training
- Answered from its training data only, with a fixed knowledge cutoff
- Couldn't know recent or private facts, and could invent details
Fits general writing and explaining.
Perplexity: look things up first
- Built around searching the web for each question
- Answers with links to its sources; slower and costlier per answer
Fits questions about specific, changing facts.
What goes wrong
3 minEach call is independent.
Your app owns the conversation and resends what each call needs.
It has patterns from training and a , not your hotels' facts.
Fetch facts from your systems and put them in the call.
Output tokens are the slow, expensive ones.
Short answers, a max_tokens cap, streaming.
Trade-offs
3 min| Model | Fits when | You pay |
|---|---|---|
| Small, cheap | Narrow tasks, high volume, tests to measure quality | Weaker on multi-step questions |
| Most capable | Hard cases, or still learning what users ask | Several times the cost, slower per token |
| Route between them | Real traffic, easy and hard mixed | A router to build and test |
- Karim · CTOWe just spent a week cutting the model bill. Build the prototype on the small model.
- Maya · Product managerWe don't know what guests will ask yet. Use the best model and learn first.
- Karim · CTOAnd then we rebuild everything for the small one?
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Which facts the concierge gets on every call. What's missing gets invented, like the rooftop pool.
- What the bot says when the answer isn't in its facts: a product decision, or you get a confident .
- How long answers may be: at ~75 tokens a second, you choose the wait, with as the cap.
Concierge v0 sends only a one-line system prompt plus the guest message, and in 40 test chats invented 3 facilities, including a rooftop pool at Hotel Alfama. Plan first. Send a system prompt with the hotel's facts loaded from our hotels table and an instruction to answer only from them and say it doesn't know otherwise. Resend previous turns from our chat history table. Cap answers at 150 tokens. Don't change the model or temperature. Rerun the 40 chats and list answers mentioning anything not in the facts.
- One full request as sent to the API contains the facts; if not, the model can't use them.
- The rerun shows the pool, shuttle and spa turned into "I don't know" answers.
- A question outside the facts ("Is there a gym?") gets an honest admission.
- It sets temperature to 0 and says hallucinations are fixed.
- It hard-codes hotel facts in the prompt text instead of loading them from the database.
- It tests with 2 chats and reports "no more hallucinations".
Check yourself
2 minNext episode: Maya asks whether the concierge could also guess which bookings will be no-shows, and Lina says that's a different kind of machine learning.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Lina debugs Monday's three bugs
Monday's 40 test chats showed three problems: an invented pool, a forgotten guest name, and answers that took 8 s. Explain each one from how an LLM works, and fix it.
- 1Read what the model actually saw
Log the exact request for each bad chat. For the pool: two lines, a one-sentence system prompt and the question. No hotel facts.
Why: The model can only use what's in the call. Reading the payload ends the guessing in a minute.
- 2Ground the invented facts
Add Hotel Alfama's real facts to the call, and a rule for gaps.
textSystem: You are StayHub's concierge for Hotel Alfama, Lisbon. Answer only from the facts below. If they don't say, reply "I'm not sure, let me check with the hotel." Facts: breakfast 7:00–10:30; no pool; parking €18/night.Why: With the true fact in the input, the most plausible continuation is now the true one.
- 3Resend the conversation
The app stores each chat and sends the earlier turns with every new message.
Why: Calls are . The model "remembers" Aiko only if her first message is in this payload, and we pay for those tokens each time.
- 4Shorten what it writes
Ask for 2–3 sentence answers, set
max_tokensto 150, and stream tokens to the screen as they arrive.Why: Output comes one token at a time, about 75 a second. A 600-token essay takes 8.5 s; an 80-token answer about 1.6 s, and streaming shows the first word after 0.5 s.
- 5Rerun the 40 chats
Same 40 test questions: 0 invented facilities, names kept, median answer 1.8 s.
Why: A fix you haven't rerun is a hope. The 40 chats become our first small test set.
| What | Rule of thumb | Engineering consequence |
|---|---|---|
| Tokens per English word | ~1.3 (≈ 4 characters per token) | 1,000 words ≈ 1,300 tokens. Other languages often need more. |
| Input tokens | Read in one parallel pass | Cheaper per token, but long prompts add cost and delay on every call. |
| Output tokens | One pass each, one after another | Usually priced several times higher. Output length decides the wait. |
| Time to first token | A few hundred ms to a few seconds | Stream the answer so guests see progress. |
| Context window | Model-dependent, tens of thousands of tokens and up | A hard limit on input + output per call. |
Interactive simulation
Visual diagram
The lifecycle of one chat message. Click a component, or start the walkthrough.
Analogy
- A brilliant improv actor on the phoneCan continue any text in the right styleAn LLMPredicts the next token very well
- A brilliant improv actor on the phoneRemembers nothing from the last callAn LLMEach call is stateless
- A brilliant improv actor on the phoneYou read them the transcript each timeAn LLMYour app resends the chat history
- A brilliant improv actor on the phoneSpeaks word by word, out loudAn LLMGenerates one token at a time
- A brilliant improv actor on the phoneImprovises a policy they never sawAn LLMA hallucination
- A brilliant improv actor on the phonePaid per word, heard and spokenAn LLMBilled per input and output token
Where the analogy breaks: an actor understands the world and can say "I don't know". A model has no built-in sense of truth. It only knows which text tends to follow which.
Check your understanding
Practice ladder
What can the model know on turn 3?
Scenario Warm-upHere's the exact request the concierge app sends on a guest's third message:
{
"system": "You are StayHub's concierge for Hotel Alfama. Facts: parking €18/night; no pool.",
"messages": [
{ "role": "user", "content": "Can I bring my dog?" },
{ "role": "assistant", "content": "Let me check that with the hotel." },
{ "role": "user", "content": "And is parking there free?" }
]
}The guest's first message, "Hi, I'm Kenji, arriving Friday", was dropped by a bug. Trace what the model can and can't know when it answers, and predict its reply.
Finish Lina's token estimate
Estimation PracticeLina is estimating the cost of the concierge test phase. Fill in steps 3 and 4.
- 2,000 test conversations a week, 6 turns each.
- System prompt with hotel facts: 800 tokens. Each guest message: 50. Each reply: 150. The full history is resent each turn.
- Illustrative prices: €1 per 1M input tokens, €5 per 1M output tokens.
- Per turn, input = 800 + earlier turns (200 each) + 50.
- Turns 1–6 input: 850, 1,050, 1,250, 1,450, 1,650, 1,850.
- Per conversation, input and output tokens: ___
- Per week, in euros: ___
Three bug reports from Porto
Debugging PracticeThe concierge is now tested at Hotel Ribeira in Porto. Three bug reports arrive:
- "It told me the hotel has a free airport shuttle." (It doesn't.)
- "I said I'm vegetarian, and two messages later it recommended the steak dinner."
- "It took 10 seconds to answer 'what time is checkout?'"
For each one, name the LLM property that explains it and the fix.
Can the concierge quote prices?
Decision StretchMaya: "Guests keep asking the concierge 'what's the cheapest hotel in Porto this weekend?' Can it just answer?"
Options on the table: - A. Let the model answer from what it knows. - B. Have the concierge say it can't quote prices and link to the search page. - C. Later: fetch live prices from StayHub's search for each such question and give them to the model.
Write a short decision for this month: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain to a hotel manager what actually happens when a guest sends a message to the AI assistant, and why it sometimes makes things up.