Skip to content

AI Infrastructure

Gateways, routing, caching and GPUs: running models in production.

AI gatewaymodel routingfallbackrate limitingtoken trackingprompt cachingsemantic cachingGPU inferencebatch inference

Lessons

  1. 30.1AI gateways and model routingInstead of every service calling AI providers directly, send all calls through one front door that picks the model, tracks the cost and handles outages. 30 min AI cost simulator
  2. 30.2Prompt caching, response caching and semantic caching"Caching" means three different things in AI apps. Each one saves more money than the last, and each one is riskier. 30 min AI cost simulator
  3. 30.3API models vs self-hosted open modelsRent an AI model by the word, or run your own on your own machines? It's mostly a question of how busy you can keep them. 30 min AI cost simulator

On the job: try it on your own project

Ask Claude Code to find every place your code calls an LLM API: which models, which keys, what timeouts, what fallbacks, and whether prompt caching is used. Write a one-page AI infrastructure note: whether you need a gateway for routing, rate limits and token tracking, where caching would save money, and whether self-hosting a model makes sense now.

Can you do this now?

0/4. Anything you can't tick yet: reopen that lesson's practice ladder.