Skip to content
Episode 133 · The room the AI called clean

Vision models: a hotel agent that reads room photos

Tuesday, 8:40. Yesterday StayHub's new AI room check marked 297 of Hotel Lumen's 300 room photos "ready", and 11 guests found stained sheets or used glasses. The hotel wants to know if they can still trust it.

An AI model can look at a room photo and tell you if the bed is made. Here's what it sees well, where it fails, and when a human decides.

30 min advancedVision model capabilities and limitsStructured findings from imagesPrivacy of image inputsStructured output from LLMsValidating model output before actingConfidence thresholds and escalation to humans
  1. Problem

    A 120-room hotel gets 300 housekeeping photos a day, guests send photos of damage or dirty rooms, and listings need accurate amenity tags. Staff can't look at everything carefully.

  2. Why the simple system fails

    Asking a vision model 'is this room OK?' returns confident prose: it misses a hair on a pillow, counts three towels as four, claims a sea view from a picture of a car park, and nobody can tell which answers to trust. Charging a guest for 'damage' it imagined is a lawsuit.

  3. Engineering concept

    Ask narrow visual questions, demand structured findings with evidence and an explicit 'unclear' option, validate them, derive confidence from checks rather than the model's tone, and send high-impact or uncertain cases to a person.

  4. Technology

    Vision-language model via the AI gateway, image preprocessing (resize, crop, EXIF strip), JSON-schema structured output, rules-based validation, a review queue, and an eval set of labelled photos.

  5. Trade-offs

    More automation saves staff time but raises the cost of each silent mistake. Higher resolution and multiple crops raise accuracy and token cost. Storing photos helps audits and evals but increases privacy exposure.

01

The problem

2 min
Tuesday, 8:40. Day two of the "Room Ready" pilot at Hotel Lumen, Lisbon.
  1. Maya · Product manager
    Yesterday the AI marked 297 of 300 room photos ready. 11 guests complained: stained sheets, used glasses.
  2. You · Engineer at StayHub
    So the model can't see stains?
  3. Lina · Senior engineer, your mentor
    Our prompt was "Is this room ready? Answer yes or no." What do you think went wrong?
02

The big idea

4 min
Why the AI said every room was ready
The big idea

Treat a vision model as a fast junior inspector: narrow questions, an allowed "unclear", and code, not the model's tone, decides what's trusted.

Change the question, not the model. Then let the cost of a mistake decide how much you automate.

03

How it works

5 min

A cuts a photo into and answers in text. It's good at recognising and describing common things, unreliable at counting, measuring and tiny details, and sounds equally sure either way.

One photo, end to end
  1. 1
    Guided capture

    The app tells the housekeeper where to stand: whole bed in frame, lights on.

  2. 2
    Clean the file

    Strip hidden location data (), keep the original private, make a resized copy.

  3. 3
    Ask for structured findings

    Structured findings: one answer per checklist item (ok, issue or unclear) with a line of evidence, instead of a paragraph.

  4. 4
    Code validates

    Schema valid? Evidence present? Photo quality good? Any critical item (stains, rubbish) not ok?

  5. 5
    Route by doubt and impact

    Confidence routing: code decides from checks it can verify. Clean results mark the room ready; doubt or critical items go to a supervisor; money always to a person.

04

Watch a decision being made

4 min
Worked example

Lina and you redesign the room check

Make wrong "ready" answers rare and send doubtful rooms to a supervisor, without reviewing every photo.

  1. 1Split the question into a checklist

    Bed made, towels, bedding stains, visible rubbish, floor clear. Mark stains and rubbish critical.

    Why: A narrow question has a visible answer; "ready" blurs many small checks together.

  2. 2Allow "unclear", demand evidence

    Each item returns ok, issue or unclear plus evidence; photo quality is its own field.

    Why: If doubt has nowhere to go, the model guesses.

  3. 3Route in code

    Rules a person can read:

    ts
    if (check.photoQuality !== "good") return { route: "retake" };
if (critical.some((i) => i.status !== "ok")) return { route: "review" };
if (check.items.some((i) => i.status === "unclear")) return { route: "review" };
if (check.items.some((i) => i.status === "issue")) return { route: "housekeeping" };
return { route: "auto_ready" };

    Why: A model's own "0.95" is poorly calibrated; wrong answers sound just as sure.

  4. 4Measure on labelled photos

    Run v2 on 300 photos supervisors already labelled; audit 200 auto-accepted rooms weekly.

    Why: Without labels, "it seems better" is a feeling.

The whole thing in one line: About 78% automatic, 18% reviewed, 4% sent back: roughly 27 minutes of supervisor time a day, instead of 2.5 hours checking everything.
05

In real life

3 min
Two well-known products, two choices

Airbnb: automate photo tagging

  • Has publicly described sorting listing photos by room type with computer vision
  • A wrong tag is a small annoyance at huge volume

Fits when mistakes are cheap.

Be My Eyes: a human one tap away

  • "Be My AI" describes photos for blind and low-vision users
  • Publicly keeps the option to call a sighted volunteer

Fits when a silent mistake can hurt someone.

06

What goes wrong

3 min
Trusting counts and measurements

Models miscount beyond a few objects and guess sizes or sea views.

Instead

Crop and compare with expected counts; take facts from property data.

Using self-reported confidence

A wrong answer can say 0.95.

Instead

Build confidence from checks, calibrated on labelled photos.

Sending raw photos anywhere

Faces, passports and GPS coordinates; text in a photo can also be an injection.

Instead

Strip EXIF, limit retention and access, treat image text as untrusted.

07

Trade-offs

3 min
OptionFits whenCosts
Full automationCheap mistakes: listing tagsSilent misses
Confidence-routed reviewCostly but not catastrophic: housekeepingA review team; drifting rules
Model assists, human decidesMoney, safety, legal: damage feesLeast time saved
Thursday, 11:00. Room Ready is about to roll out to 40 hotels.
  1. Karim · CTO
    18% review is 54 photos a day per hotel, times 40. Auto-accept everything except clear issues.
  2. Lina · Senior engineer, your mentor
    Then every unclear stain becomes "ready" again.
  3. Karim · CTO
    Then lower the rate with better photos, and review only critical items.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • The checklist and its answers (ok, issue, unclear, with evidence), not "Is this room ready?".
  • The routing rules, in code: what auto-passes, what a supervisor sees, what goes back.
  • What the model never decides alone (any charge) and never gets asked (sea view, room size).
Tell Claude Code
Rebuild the photo check in `src/room-ready/` for Hotel Lumen; "ready? yes/no" passed 297 of 300 photos. Plan first. Checklist (bed made, bedding stains, rubbish, used glasses, towels), each ok, issue or unclear with evidence, validated by a JSON schema. Routing rules in code: critical not ok or anything unclear to the supervisor queue, other issues to housekeeping. Strip EXIF. Nothing may charge a guest. Run the 300-photo eval set; list every wrong auto-accept.
Check before you accept
  • Shares near 78 / 18 / 4%, and you open five wrong auto-accepts yourself.
  • Routing rules are readable code, not prompt instructions.
  • Nothing in the diff touches payments or fees.
Red flags
  • The model's own confidence: 0.93 used as the auto-accept threshold.
  • No unclear option in the schema.
  • The eval set replaced by a few new photos the prompt happens to pass.
09

Check yourself

2 min
Q1/3Reasoning
If you ask the vision model to output a confidence score and only auto-accept results above 0.9, you have a calibrated confidence threshold.
How confident are you?

Next episode: finance is drowning in 9,000 supplier invoices a month, and the AI that reads them just booked a PO number that doesn't exist.

Finished?

Mark it complete to update your skill map.

Next: Document extraction: invoices, passports and contracts to structured data
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

From confident prose to checkable findings
Worked example

Lina and you redesign the room check

Room check v1 asks "Is this room ready? yes/no" and marked 297 of 300 photos ready, with 11 complaints. Redesign it so wrong "ready" answers become rare and the doubtful ones reach a supervisor, without reviewing every photo.

  1. 1Split the question into a checklist

    Replace "is it ready?" with narrow, visible items: bed made, towels present, bedding stains, visible rubbish, floor clear. Mark which ones are critical (stains, rubbish).

    Why: A narrow question has a visible answer. "Ready" is a judgement made of many small checks, and the model blurs them together.

  2. 2Allow "unclear" and demand evidence

    Each item returns ok, issue or unclear, plus a short evidence string. Photo quality is its own field.

    ts
    const RoomCheck = z.object({
  photoQuality: z.enum(["good", "too_dark", "blurry", "wrong_angle"]),
  items: z.array(z.object({
    item: z.enum(["bed_made", "towels", "bedding_stains", "visible_rubbish", "floor_clear"]),
    status: z.enum(["ok", "issue", "unclear"]),
    evidence: z.string().max(200),
  })),
});

    Why: If doubt has no place to go, the model guesses. Evidence makes each finding something a supervisor can check in seconds.

  3. 3Validate in code and build confidence

    Parse with the schema (), then apply rules: bad photo → retake; critical item not ok → review; issue without evidence → review; any unclear → review; any other issue (with evidence) → back to housekeeping.

    ts
    if (check.photoQuality !== "good") return { route: "retake" };
const critical = check.items.filter((i) => CRITICAL.includes(i.item));
if (critical.some((i) => i.status !== "ok")) return { route: "review" };
if (check.items.some((i) => i.status === "issue" && !i.evidence)) return { route: "review" };
if (check.items.some((i) => i.status === "unclear")) return { route: "review" };
if (check.items.some((i) => i.status === "issue")) return { route: "housekeeping" };
return { route: "auto_ready" };

    Why: Confidence comes from things we can verify. A model's own "0.95" is poorly calibrated: wrong answers often sound just as sure.

  4. 4Route by impact

    Room ready → automatic when everything is clean. Stains, rubbish, doubt → supervisor. Anything involving money (damage fees) → always a person.

    Why: The same accuracy is fine for one task and dangerous for another. The cost of a mistake sets how much we automate.

  5. 5Measure on labelled photos

    Run v2 on 300 photos supervisors already labelled. Check: of the rooms auto-marked ready, how many were really ready? Then audit 200 auto-accepted rooms every week.

    Why: Without labels, "it seems better" is a feeling. The eval set also tells us which item is weakest so we can fix that one.

The whole thing in one line: We replaced one broad yes/no with a checklist the model can answer honestly, let code decide from verifiable checks, sent doubt and money to people, and proved it on labelled photos: about 78% automatic, 18% reviewed, 4% sent back.
Visual taskTypical reliabilityDesign response
Is there a bed / bathtub / balcony?HighAutomate; spot-check
Is the bed made? Rubbish visible?Good with clear photosChecklist + unclear + capture guidance
Count pillows, towels, glassesUnreliable above a few itemsCrop the area; compare with the expected count; review mismatches
Small stains, hairs, cracksUnreliable on resized imagesHigher-resolution crops; human review for critical items
Room size in m², "sea view"Often guessedDon't ask the model; use property data
Is this photo genuine / from this room?Cannot verifyNever base charges on it alone; human review
Illustrative; measure on your own labelled photos. Reliability varies by model, resolution and photo quality.
11

Visual diagram

Click a component

Housekeeping and damage photos flowing through a vision pipeline with validation and human review.

12

Analogy

  • A remote supervisor seeing photos"Is the room fine?" gets "Looks fine!"A vision pipelineA vague prompt gives useless answers
  • A remote supervisor seeing photosA checklist with "can't see"A vision pipelineStructured questions with an unclear option
  • A remote supervisor seeing photosCan't see under the bedA vision pipelineCan't judge what isn't visible
  • A remote supervisor seeing photosCharges go to the managerA vision pipelineHumans decide anything costly

Where the analogy breaks: a supervisor who's unsure will say so. A model often won't, so you must build "unclear" into the questions and check its answers in code.

13

Check your understanding

Q1/3Decision
Which of these should go to a human reviewer? (select all that apply)

Select all that apply.

How confident are you?
14

Practice ladder

Step 1 · Trace

Trace three room checks through the validator

Scenario Warm-up

The v2 validator rules, in order: 1. photoQuality not good → retake 2. A critical item (bedding_stains, visible_rubbish) not ok → review 3. Any issue without evidence → review 4. Any unclear → review 5. Any other issue (with evidence) → back to housekeeping 6. Otherwise → auto_ready

Trace these three results:

  • Room 204: quality good; bed_made ok; towels unclear ("rail partly hidden"); bedding_stains ok; visible_rubbish ok.
  • Room 318: quality too_dark; every item ok.
  • Room 411: quality good; bed_made issue (evidence: ""); every other item ok.

For each room, say which rule fires first and where the room goes.

Step 2 · Complete

Finish Lina's damage-report flow

Design Practice

Guests can report problems in the StayHub app with up to 3 photos and a sentence ("The shower door is broken"). Lina's flow has two gaps. Fill in steps 3 and 5.

  1. Clean the upload: strip EXIF, store the original privately with a 90-day retention.
  2. Ask narrow questions: category (plumbing, electrical, furniture, cleanliness, other), visible damage (yes / no / unclear), evidence, urgency suggestion.
  3. Validate and route: ___
  4. Maintenance ticket: created with the model's draft description and the photo.
  5. Money: ___
Step 3 · Solve

40 rooms with a "sea view" that faces a car park

Debugging Practice

The listing-tagging job tagged 40 rooms at a beach hotel as "sea view". Guests complain they see a car park. The photos were taken through the window at an angle that shows a strip of blue at the edge.

Maya wants a fix this week. Explain what went wrong and design the new tagging rule for "sea view" and similar facts (room size, floor, quiet room).

Step 4 · Decide

Auto-approve small refunds from guest photos?

Decision Stretch

Maya: "Guests wait 2 days for a refund when the room isn't clean. Can the AI approve refunds under 30 € automatically when the photo clearly shows a dirty room?"

The facts: - About 60 cleanliness complaints a week across StayHub's partner hotels, average refund 22 €. - On 300 labelled complaint photos, the model's "clearly dirty" findings were right 94% of the time. - Photos can be old, from another stay, or from another hotel; the model can't tell. - Support handles each complaint in about 6 minutes today.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

15

Explain it yourself

Explain how you'd make a vision model's room-photo checks trustworthy enough for daily hotel operations.

0 words