Skip to content
Episode 132 · Four GPUs and a quote

API models vs self-hosted open models

Monday, 10:00: Karim has a quote for four GPUs at 7,200 EUR a month to run an open-weight model ourselves and "stop paying per token". The AI bill is 12,000 EUR, and a 60-hotel chain wants guest data kept in the EU. Lina: "You lead the decision."

Rent an AI model by the word, or run your own on your own machines? It's mostly a question of how busy you can keep them.

30 min advancedSelf-hosted model inferenceManaged services vs self-hostingAPI vs GPU cost crossoverInference batching (online and offline)Model quantization trade-offData residency & GDPR constraints
  1. Problem

    StayHub's AI bill is ~€12,000 a month even after routing and caching, a GPU quote looks similar, and a new enterprise customer (Riviera Group, 60 hotels) asks that guest messages are processed in the EU, not retained and not used for training.

  2. Why the simple system fails

    'Just run an open model on a GPU' ignores that a GPU costs money every hour whether busy or idle, that one request at a time wastes most of its capacity, that the best open model may be weaker at your task, and that someone now gets paged at 3 a.m. for it.

  3. Engineering concept

    Compare fully loaded cost per useful output at your real volume and utilisation, plus quality on your evals, plus privacy and latency needs, plus the operational burden. Self-hosting wins at high, steady volume or hard data constraints; APIs win almost everywhere else.

  4. Technology

    Hosted model APIs (including batch APIs and regional endpoints) vs open-weight models served by an inference server with continuous batching, quantization and GPU autoscaling, often behind the same AI gateway.

  5. Trade-offs

    APIs: variable cost, best models, no ops, data leaves your boundary. Self-hosted: fixed cost, full control and residency, cheaper at scale if utilisation is high, but you own capacity, upgrades, evals and on-call.

01

The problem

2 min
Monday, 10:00. Karim drops a quote on the table.
  1. Karim · CTO
    4 GPUs, 7,200 EUR a month. Our AI bill is 12,000. Let's run an open-weight model and stop paying per token.
  2. Maya · Product manager
    And Riviera Group, 60 hotels, wants guest messages processed in the EU, not used for training, not retained.
  3. Lina · Senior engineer, your mentor
    You lead this one. I'll check your numbers.
02

The big idea

4 min
Per token or per hour
The big idea

Compare the fully loaded cost at real utilisation against the cheapest hosted option that passes your evals, for the tasks the open model can actually do.

Taxi or your own van: the van is cheaper per trip only if you drive it all day, you pay for it while it sits still, and it can't do every trip.

03

How it works

5 min

An open-weight model publishes its weights, so you can run it on your own GPUs. APIs charge per token and nothing when idle; GPUs charge per hour, busy or not.

Deciding between an API and self-hosting
  1. 1
    Write the exact requirements

    Quality per task, latency at peak, and the exact data rule ("EU, no training, no retention"), not "privacy" in general.

  2. 2
    Evaluate the open model per task

    Golden set per task. Quantization (fewer bits per weight, to fit smaller GPUs) is a new model version: evaluate each precision.

  3. 3
    Price the cheapest option that passes

    The tasks it passes already run on the small API model: ~1,200 EUR a month.

  4. 4
    Fully load the GPUs

    Fully loaded cost: GPUs 24/7 + engineer time + monitoring. 7,200 + 4,000 + 300 ≈ 11,500 EUR a month.

  5. 5
    Check utilisation

    Utilisation is the share of paid GPU time doing useful work. The evening peak is 3× the average, so a peak-sized pool is ~30% busy.

  6. 6
    Decide per task, write a trigger

    Keep what fails evals on the API; self-host only where it clearly pays; write down when to revisit.

04

Watch a decision being made

4 min
Worked example

You decide: API or self-hosted (Lina checks)

Answer Karim with numbers, meet Riviera's data rule, and leave a rule for when to revisit.

  1. 1Evaluate per task

    The 8B model at 8-bit passes FAQ, classification, summaries; fails booking (multi-room 74% vs 90%) and refunds. 4-bit loses 6 more points on price extraction.

    Why: The model can only replace calls it handles.

  2. 2Compute break-even

    Against the small API model at ~0.0009 EUR a call:

    python
    fixed = 4 * 2.5 * 24 * 30 + 4000 + 300   # 11,500 EUR/month
break_even = fixed / (30 * 0.0009)        # ~426,000 easy calls/day

    Why: We have ~45,000 easy calls a day. GPUs bill when idle, and people are part of the cost.

  3. 3Meet the data rule simply

    Legal confirms the provider's EU regional endpoint with zero retention and no training meets Riviera's wording; the gateway routes their hotels there.

    Why: Self-host for privacy only if contracts and regional endpoints can't meet the exact rule.

  4. 4Decide and set a trigger

    Stay hosted. Revisit above ~400,000 easy calls a day, if no provider can meet a customer rule, or if an open model passes the booking slices.

    Why: A written trigger turns a monthly debate into a check anyone can run.

The whole thing in one line: Spending 11,500 to save 1,200 doesn't add up. Lina read the decision and had nothing to add.
05

In real life

3 min
Hosted models vs running your own

GitHub Copilot: hosted models

  • Publicly built on models from external providers
  • Gets new frontier models as they're released; data governed by contracts

Best models fast; depends on providers.

Apple Intelligence: its own models

  • Publicly described as running on-device and on Apple's own "Private Cloud Compute"
  • Privacy and control were the stated reasons; still integrates ChatGPT for some requests

Full control at enormous engineering cost.

06

What goes wrong

3 min
Comparing GPU rent against the whole API bill

The alternative is the cheapest hosted option for the tasks the open model passes.

Instead

Evaluate per task, then compare fully loaded cost against what it replaces.

Sizing for the average with no overflow

Above capacity a queue only grows; GPUs take minutes to start.

Instead

Size for peak, autoscale early, overflow to the API via the gateway.

Self-hosting "for privacy" without checking API terms

Regional endpoints and zero-retention terms often meet the exact rule.

Instead

Write the exact rule; check provider terms with legal first.

07

Trade-offs

3 min
OptionFits whenCosts
Hosted APIVariable volume, quality-sensitive tasks, no GPU teamGrows with usage; third-party processing
Self-hosted open modelHigh steady volume on a task it passes, a team to run itIdle capacity, on-call, slow scaling
Hybrid behind a gatewayOne high-volume simple task plus a complex long tailTwo stacks to evaluate
Monday, 15:00. Karim reads your analysis.
  1. Karim · CTO
    Providers can change prices under us. Owning capacity is strategic.
  2. Omar · On-call SRE
    Owning capacity means 3 a.m. pages for drivers and a server we've never run. We have no one.
  3. Karim · CTO
    We'll never have someone if we never start.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • What you compare against: today's routed and cached API bill (~12,000 EUR), not the old all-large-model one.
  • Which tasks may move: only slices that pass evals at the exact precision you'll deploy.
  • Capacity: sized for the evening peak (~3× average), with API fallback when the GPU queue grows.
Tell Claude Code
Prepare, don't deploy, a self-hosting pilot for easy concierge FAQs behind `services/ai-gateway/`. Plan first. Only the FAQ route moves; 8B open model at 8-bit; size the pool for the evening peak using last month's hourly traffic from the usage logs (show me); fall back to the hosted API when queue time passes 2 s or the pool errors. Write a load test replaying the 20:00 peak, and a cost sheet at our real volume including ops time.
Check before you accept
  • The cost sheet includes GPU hours, engineer time and utilisation.
  • The peak number comes from measured hourly traffic, not an assumed average.
  • Stopping the staging pool sends traffic to the API.
Red flags
  • Pool sized from the average because "autoscaling will handle peaks".
  • Self-hosting compared with the old 48,000 EUR bill.
  • 4-bit chosen to save memory without the price-extraction evals.
09

Check yourself

2 min
Q1/3Scenario
A hosted model costs €0.005 per call. Your fully loaded self-hosted cost (GPUs, ops time, monitoring) is €12,000 a month, and evals show equal quality. Above roughly how many calls per day does self-hosting become cheaper (30-day month)?
calls/day
How confident are you?

Next episode: a new product, Room Ready. On its first day the AI calls 297 of 300 housekeeping photos "ready", stained sheets included.

Finished?

Mark it complete to update your skill map.

Next: Vision models: a hotel agent that reads room photos
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

The numbers behind the decision
Worked example

You decide: API or self-hosted (Lina checks)

Answer Karim's proposal with numbers, meet Riviera's data rule, and leave a clear rule for when to revisit.

  1. 1Write down the requirements

    Quality: each task must pass its golden-set slice. Latency: concierge p95 under 3 s at the 20:00 peak. Data: Riviera's guest messages processed in the EU, no training, no retention.

    Why: "Privacy" and "cheaper" are too vague to decide anything. Exact requirements often turn out to be met by simpler options.

  2. 2Evaluate the open model per task

    Run the open 8B model at 8-bit and 4-bit through the eval suite. 8-bit passes FAQ, classification, summaries; fails booking (multi-room 74% vs 90%) and refunds. 4-bit loses 6 more points on price extraction.

    Why: The model can only replace calls it handles. Quantization is a model version, so it gets its own eval run.

  3. 3Price the alternative it would replace

    The tasks it passes run today on the small API model: ~45,000 calls/day × ~0.0009 EUR ≈ 1,200 EUR/month.

    Why: Compare against the cheapest hosted option that passes, not against the whole bill or the most expensive model.

  4. 4Fully load the self-hosted cost

    4 GPUs × 2.5 EUR × 24 × 30 = 7,200; + ~4,000 for engineer time and monitoring; + 300 overhead ≈ 11,500 EUR/month, sized for a peak 3× the average (~30% average utilisation).

    python
    fixed = 4 * 2.5 * 24 * 30 + 4000 + 300      # 11,500 EUR/month
break_even = fixed / (30 * 0.0009)           # ≈ 426,000 easy calls/day

    Why: GPUs bill when idle, and people are part of the cost. Utilisation turns a cheap-looking quote into an expensive one.

  5. 5Meet the data rule the simple way

    Legal confirms our provider's EU regional endpoint with zero retention and no training meets Riviera's wording. The gateway routes Riviera's hotels there.

    Why: Self-hosting for privacy only makes sense if contracts and regional endpoints can't meet the exact rule.

  6. 6Decide and set a trigger

    Stay hosted. Revisit when easy-task volume passes ~400,000 calls/day, when a customer rule can't be met by any provider, or when an open model passes the booking slices.

    Why: A written trigger turns a recurring debate into a check anyone can run.

The whole thing in one line: You compared the fully loaded GPU cost (~11,500 EUR) with the ~1,200 EUR of API calls the open model could actually replace, met Riviera's rule with a regional zero-retention endpoint, and wrote down when the answer would change.
Model size16-bit weights8-bit4-bitNotes
8B parameters~16 GB~8 GB~4 GBFits one mid-range GPU with room for the KV cache
30B parameters~60 GB~30 GB~15 GBOne large GPU when quantized
70B parameters~140 GB~70 GB~35 GBSeveral GPUs at 16-bit; one large GPU at 4-bit
Weights only. Long contexts and many concurrent requests add tens of GB of KV cache. Quality loss from quantization varies: measure it.
DimensionHosted APISelf-hosted open model
Cost shapePer token; zero when idlePer GPU-hour; paid when idle
Best qualityFrontier proprietary modelsBest open-weight models; often strong on narrow tasks
ScalingInstant (within rate limits)Minutes per replica
Data controlProvider terms, regional endpoints, retention settingsStays in your network
Ops burdenKeys, limits, fallbacksGPUs, drivers, inference server, autoscaling, on-call
11

Interactive simulation

12

Visual diagram

Click a component

A hybrid setup (StayHub's "if triggered" stage, not today's): the gateway sends steady high-volume work to self-hosted GPUs and hard tasks to a hosted API.

13

Analogy

  • Getting around a cityTaking taxis, paying per rideRunning AI modelsHosted API, paying per token
  • Getting around a cityOwning vans, even when parkedRunning AI modelsSelf-hosted GPUs, paid even when idle
  • Getting around a cityHiring mechanicsRunning AI modelsRunning the machines yourself
  • Getting around a cityFilling every seat in the vanRunning AI modelsBatching many requests together
  • Getting around a cityA smaller, lighter vanRunning AI modelsQuantization: a smaller model copy

A delivery company with steady routes buys vans. A consultant who travels twice a month takes taxis. Many companies do both. Where the analogy breaks: you can buy any van, but the very best AI models usually aren't available to run yourself.

14

Check your understanding

Q1/3Reasoning
Roughly how many GB of GPU memory do the weights of a 30-billion-parameter model need at 8-bit (1 byte per parameter)?
GB
How confident are you?
15

Practice ladder

Step 1 · Trace

Trace a cost comparison

Estimation Warm-up

The nightly review-summary job: 6,000 reviews/day, ~1,500 input + 200 output tokens each. Options: - Small API model: 0.20 / 0.80 EUR per million input/output tokens; batch API at 50% off. - Self-hosted: 1 GPU at 2.5 EUR/hour, but only 4 hours a night (started and stopped by a scheduler), + 1,500 EUR/month of engineer time.

The open model passes the review-summary eval. Trace the monthly cost of each option and say which is cheaper.

Step 2 · Complete

Finish the decision memo

Decision Practice

Your memo to Karim has two empty sections. Fill in Data and Revisit when.

  1. Quality: the open 8B model passes FAQ, classification and review summaries; fails booking and refund slices.
  2. Cost: self-hosted ~11,500 EUR/month fully loaded vs ~1,200 EUR/month of API calls it could replace.
  3. Latency: peak is 3× average; a peak-sized pool averages ~30% utilisation.
  4. Data: ___
  5. Decision: stay on hosted APIs through the gateway.
  6. Revisit when: ___
Step 3 · Solve

A fine-tuned model for hotel tags

Design Practice

The revenue team wants to tag every review with 40 hotel-specific labels ("thin walls", "lift slow", "great rooftop"). The small API model gets 78% on the tagging eval; a fine-tuned open 8B model gets 91%. Volume: 6,000 reviews/day, all nightly. Design the deployment: where it runs, sizing, fallback, evals, and the cost comparison you'd present.

Step 4 · Decide

A customer who forbids any third party

Decision Stretch

A new prospect, a government-owned hotel group (40 hotels), requires that guest messages are processed only on infrastructure StayHub operates: no third-party model provider at all, even in the EU. They'd be 15% of concierge traffic. The best open model passes FAQ and classification but fails our booking slice.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain to a CFO why 'open models are free' doesn't mean self-hosting is cheaper, and when it actually is.

0 words