Episode 134 · The PO number nobody wrote

Document extraction: invoices, passports and contracts to structured data

Wednesday, 16:10. Finance has found 37 invoices this month booked against purchase-order numbers that don't exist. The AI that reads StayHub's 9,000 monthly invoices made them up, in perfectly valid JSON.

Turning invoices, passports and contracts into clean data: the AI reads, your code checks, and people only look at what doesn't add up.

30 min advancedOCR pipeline vs native document understandingSchema and business-rule validation of extracted dataField-level human reviewEvaluating extraction accuracyStructured output from LLMsValidating model output before acting

Problem
StayHub's finance team receives 9,000 invoices a month from partner hotels and suppliers as PDFs and phone scans, hotels scan guest passports at check-in for legal registration, and sales signs hundreds of corporate rate contracts a year. People retype all of it.
Why the simple system fails
Classic OCR gives you characters, not meaning: it can't tell the invoice total from the subtotal. A plain 'extract to JSON' prompt returns valid-looking JSON with a misread digit or an invented PO number, and nothing stops it reaching the accounting system.
Engineering concept
Extraction is a pipeline: get the best text or image, extract into a strict schema, validate with types plus business rules (sums, checksums, cross-references), route only failing fields to a human, and measure field-level accuracy on a golden set.
Technology
PDF text layers, OCR engines, multimodal models with native document input, structured output with JSON Schema, Pydantic/Zod validation, check-digit and arithmetic rules, a review UI with source highlighting, and field-level eval scripts.
Trade-offs
Native document models handle varied layouts but cost more and can hallucinate plausibly; OCR plus rules is cheap and predictable but brittle on new layouts. More automation raises straight-through rates and the cost of each undetected error.

The problem

2 min

Wednesday, 16:10. A week after Room Ready went to 40 hotels.

Maya · Product manager
Finance found 37 invoices this month booked against PO numbers that don't exist. The AI reads all 9,000 a month.
You · Engineer at StayHub
So the model misread the PO field?
Lina · Senior engineer, your mentor
You lead this one. Open three of those invoices and look for a PO.

The big idea

4 min

Valid JSON, invented data

The big idea

Structured output guarantees shape, not truth. The model reads; your code checks the values.

Extraction is a pipeline, not a prompt: pick the best reader, allow "not on the document", check values with rules, and send only the failures to a person.

How it works

5 min

One invoice, end to end (v2)

1
Pick the reader
Born-digital PDF (about 70%)? Use its text layer: exact and nearly free. Scan? (pixels to characters, no meaning) or a document understanding model that reads layout, tables and labels.
2
Fill a strict schema
Fields that may not be printed are null, never guessed (). Each field records its page and box.
3
Run business checks
Business checks test values in code: lines add up, net + VAT = total, IBAN checksum and vendor master match, PO exists, not already paid.
4
Review fields, not documents
Only failing fields go to a clerk, with the reason and the source region highlighted. Ten seconds, not five minutes.
5
Measure both numbers
Straight-through processing: documents that pass every check with no human touch. Report that rate and the errors that slip past it.

Watch a decision being made

4 min

Worked example

You rebuild the invoice pipeline (Lina checks)

Catch invented or misread values before accounting, without a clerk re-checking all 9,000 invoices.

1Choose the reader per document
Text layer when present; the document model only for scans.
Why: Text already in the file is exact. Lina: "And it cuts the model bill by about two thirds."

2Allow missing values

Nullable fields with a source per field:

python

po_number: str | None      # None if not printed: never guess
iban: str | None
sources: dict[str, Source]  # field -> page + box

Why: A required field with no answer is an order to invent one.

3Check values, return failing fields
Lines sum to 1,240.50 but net total read as 1,746.50? Only net_total fails.
Why: Arithmetic catches a 1 read as a 7; a lookup catches an invented PO. No human needed.
4Measure per field
Golden set of 500 verified invoices: accuracy per field, invented and missed values separately. Audit 100 straight-through invoices a week.
Why: "97% accurate" can hide the IBAN field at 91%.

In real life

3 min

Two well-known approaches

Airport e-gates: machine-made parts

Passports carry a machine-readable zone with check digits, and a chip
Can't read it? You go to the officer; no guessing

Fits documents designed for machines.

Expensify: messy receipts, humans behind

Endless layouts, crumpled and photographed
Has publicly described SmartScan with human review for receipts it can't read reliably

Fits endless layouts where errors must stay low.

What goes wrong

3 min

Making every field required

The model fills them anyway: invented POs, guessed dates.

Instead

Nullable fields, "never guess", and an eval metric for invented values.

Treating valid JSON as correct data

A 7 misread for a 1 parses perfectly.

Instead

Business rules: sums, check digits, master data, duplicates.

Updating master data from documents

Documents can be forged.

Instead

Bank changes need a call-back to a known contact and a second approver.

Trade-offs

3 min

Reader	Fits	Fails by
OCR + templates	Few fixed layouts, passport zones	Breaking silently on new layouts
Document model	Many varied layouts and scans	Plausible-but-wrong values
Hybrid	Mixed digital PDFs and scans	Two paths to test and monitor

Thursday, 14:00. Month-end close is in six days.

Karim · CTO
2,700 reviews a month for 3 clerks. Loosen the rules: 85% straight-through.
Lina · Senior engineer, your mentor
The audit says loose rules let ~1.2% slip. Over 90 errors a month.
Karim · CTO
Most are small amounts. Cheaper than three more clerks.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which fields are nullable: anything that may not be printed comes back null, never invented.
The business checks that gate straight-through, and what never goes straight through: an IBAN that differs from the vendor master.
Your error budget (~0.5% of 6,300, about 32 a month), measured by a weekly audit of 100.

Tell Claude Code

Rebuild invoice extraction in `src/finance/extraction/`; required fields produced 37 invented POs. Plan first. Text layer when present, images for scans; po_number, vat_number, due_date nullable, "null if not printed"; each field returns its source region; validators check sums, IBAN checksum, vendor master match, duplicates; failing fields go to review alone; an IBAN differing from the master always goes to review. Report the 500-invoice golden set per field: correct, wrong, missed, invented.

Check before you accept

"Invented where truth is null" for po_number is near zero, reported separately.
A known vendor's invoice with a changed, valid IBAN stops in review.
The review UI shows only failing fields with highlighted sources.

Red flags

Every field stays required "so the import doesn't break".
Self-reported confidence decides straight-through instead of business checks.
Accuracy reported on a few born-digital PDFs, skipping the 30% scans.

Check yourself

2 min

Q1/3Debugging

Accounting reports invoices booked against PO numbers that don't exist. The schema marks po_number as a required string. Most of these invoices have no PO printed at all. What is the root cause and fix?

How confident are you?

Next episode: StayHub's new phone concierge answers night calls correctly, after 3.5 seconds of silence each time, and almost a third of callers hang up.

Finished?

Mark it complete to update your skill map.

Next: The voice agent pipeline: microphone to speaker in under a second

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Checks that need no human

Worked example

You rebuild the invoice pipeline (Lina checks)

v1 books every invoice the model returns as valid JSON. 37 invented POs this month. Rebuild it so invented or misread values are caught before accounting, without a clerk re-checking all 9,000 invoices.

1Choose the reader per document
Check if the PDF has a usable text layer (about 70% do). Use it; send only scans to the document model.
Why: Text already in the file is exact. Turning it into an image and reading it back costs more and adds misreads. Lina: "Right, and it cuts the model bill by about two thirds."

2Allow missing values

Fields that may not be printed are nullable, and the prompt says "null if not printed; never guess". Each field records its page and box.

python

class Invoice(BaseModel):
    supplier_name: str
    invoice_number: str
    po_number: str | None      # None if not printed: never guess
    iban: str | None
    line_items: list[LineItem]
    net_total: Decimal
    total: Decimal
    sources: dict[str, Source]  # field -> page + box

Why: A required field with no answer is an order to invent one. The source box lets a reviewer check a value in seconds.

3Check values, not just shape
After parsing, run business checks that return failing fields: lines sum to net, net + VAT = total, IBAN checksum, supplier in the vendor master, IBAN equals the vendor master's, invoice not already paid, PO (if any) exists.
Why: These checks catch most misreads and inventions with no human. A 1 read as a 7 breaks the sum; an invented PO fails the lookup.
4Review fields, not documents
Empty failure list → straight through. Otherwise the clerk sees only the failing fields, the reason, and the highlighted source region.
Why: Re-checking 20 correct fields to find 1 wrong one makes reviewers tired and careless. Lina: "This is the part people skip, and then they rubber-stamp."
5Measure field by field
Run the pipeline on a golden set of 500 human-verified invoices. Track accuracy per field, hallucinated values (value where truth is null) and missed values separately. Audit 100 straight-through invoices a week.
Why: A single "97% accurate" hides the IBAN field at 91% and the supplier whose scans fail. The audit is the only honest estimate of errors that reach the books.

Reading method	Best for	Cost	Typical failure
PDF text layer	Born-digital PDFs	~free, exact characters	Text order scrambled in complex layouts; still needs field mapping
OCR + templates/rules	Few fixed layouts, passport MRZ, forms	Low	Breaks on new or changed layouts; no notion of meaning
OCR + LLM on text	Varied layouts, text-heavy docs	Medium	Loses table structure; mixes up columns
Native document model	Varied layouts, scans, tables, stamps	Higher (image tokens per page)	Plausible but wrong values; invented optional fields

Mix methods per document type. Use deterministic structure (text layers, MRZ, barcodes) wherever it exists.

Visual diagram

Invoice inbox

Document store

Classify + prepare

Document model

Validator

Review UI

Accounting system

Golden set + metrics

Click a component

An invoice extraction pipeline. Start the walkthrough to follow one scanned invoice.

Analogy

An accounts officeDocument extraction

An accounts officeA fast typist who doesn't know accountingDocument extractionOCR copies text, no meaning
An accounts officeA clerk who "fills in" what they expectDocument extractionA model inventing plausible values
An accounts officeThe auditor with a calculatorDocument extractionValidation rules in code
An accounts officeCalling the clerk over for two fieldsDocument extractionField-level human review

Where the analogy breaks: a clerk gets tired by the afternoon. The pipeline doesn't, but it also never notices on its own that something feels off. Only your checks catch that.

Check your understanding

Q1/3Design

Order the steps for processing an incoming invoice:

1.Post the verified record to accounting
2.Store the original document (encrypted)
3.Run schema and business-rule checks
4.Extract fields into the schema, with source locations
5.Send only failing fields to review with highlights
6.Check for a usable text layer; otherwise render page images

How confident are you?

Practice ladder

Step 1 · Trace

Which fields fail?

Scenario Warm-up

The model extracted this from a scanned invoice:

{
  "supplier_name": "Padaria Alfama",
  "supplier_vat_id": "PT509112233",
  "invoice_number": "PA-0912",
  "po_number": null,
  "iban": "PT50 0035 0000 1234 5678 9012 3",
  "line_items": [
    { "description": "Bread rolls", "quantity": 400, "unit_price": 0.30, "amount": 120.00 },
    { "description": "Croissants", "quantity": 250, "unit_price": 0.60, "amount": 150.00 }
  ],
  "net_total": 270.00,
  "vat_total": 16.20,
  "total": 268.20
}

Facts: Padaria Alfama is in the vendor master with the same IBAN, and the IBAN checksum passes. Invoice PA-0912 hasn't been paid. Trace the business checks and say which fields go to review, and why.

Step 2 · Complete

Finish the field-level eval

Code Practice

Lina's eval compares each predicted field with the human-verified value on the golden set. Two branches are missing. Fill them in so the eval counts hallucinated and missed values separately.

def score_field(f, truth, pred, c):
    t, p = norm(f, truth.get(f)), norm(f, pred.get(f))
    if t is None and p is not None:
        ___                            # (1)
    elif t is not None and p is None:
        ___                            # (2)
    elif t is not None:
        c[f"{f}.present"] += 1
        if t == p:
            c[f"{f}.correct"] += 1

Step 3 · Solve

Passports at express check-in

Design Practice

StayHub's new express check-in lets guests scan their passport in the app the day before arrival. Hotels must register guests with the authorities (name, nationality, document number, birth date, expiry).

Design the extraction: which reader, which checks, what happens when it fails, and how the data is handled.

Step 4 · Decide

Pick the straight-through threshold

Decision Stretch

Three validation configs, measured by audit on StayHub's 9,000 invoices a month:

Config	Straight-through	Errors among straight-through
Loose	85%	1.2%
Medium	70%	0.5%
Strict	50%	0.1%

A slipped error costs about 150 € on average to find and fix later.
A field-level review costs about 1 € of clerk time.
The 3 clerks can review at most about 3,000 invoices a month without delaying month-end.

Write a short decision: "We chose X over Y because…, accepting the risk of…", with the numbers.

Explain it yourself

Explain how you would design an invoice extraction system that finance can trust.

0 words