Proxies, load balancers, gateways and firewalls
Tuesday, 10:15: one travel agency's script sends 40,000 requests a minute to StayHub's partner API, and search for ordinary guests slows from 300 ms to 2.5 s. You lead the response.
Between your users and your servers sit a few helper boxes. Here's what each one does, and the one question that explains them all.
- Problem
A hotel platform exposes a website, a public API, a partner API and an admin tool, and must block attacks, spread load and keep databases unreachable from the internet.
- Why the simple system fails
Exposing app servers directly means every server handles TLS, auth, rate limits and attack traffic itself, you can't add or remove servers without clients noticing, and one open port can expose a database to the whole internet.
- Engineering concept
Put specialised intermediaries in the path. Reverse proxies speak for your servers, forward proxies speak for your clients, firewalls decide who may connect, and each layer only does what it can see.
- Technology
Nginx/Envoy-style reverse proxies, L4 and L7 load balancers, API gateways, network firewalls and security groups, WAFs, VPNs, CDNs.
- Trade-offs
Every intermediary adds latency, cost, configuration and a failure point, and can become a bottleneck or a single point of failure. Each must earn its place.
The problem
2 min- Omar · On-call SRE/partner/availability: 40,000 requests a minute from one IP. All partners together normally send about 3,600.
- Maya · Product managerGuest search p95: 300 ms → 2.5 s. TravelNest says their sync script has a bug.
- Karim · CTOBlock their IP at the firewall. Now.
The big idea
4 minA box can only decide on what it can see. Put each rule in the first box that sees what the rule needs.
Like a hotel entrance: the doorman sees which door you use, reception reads what you ask for, the business desk checks your company badge. Limiting one partner needs the badge: the API key.
How it works
5 min- 1CDN edge: a reverse proxy
A receives requests for your servers and passes them on. The CDN ends TLS near the caller, caches, and absorbs junk floods. Its mirror, a , speaks for our servers going out.
- 2WAF rules
A reads the decrypted request for known attack patterns and coarse per-IP limits.
- 3API gateway (new)
An is an L7 proxy that knows who's calling: checks the key, counts requests per key, refuses the excess with
429andRetry-After. - 4Load balancer: L4 vs L7
An box sees addresses, ports and connections. An box reads the HTTP request (path, headers) and can route by path, to a separate partner pool.
- 5Firewall rules
A allows connections by IP and port. Services accept 8080 only from the load balancer; the database 5432 only from services.
Watch a decision being made
4 minYou design the partner edge; Lina only checks
Make sure no partner, buggy or not, can slow guest search again, without blocking legitimate partner traffic.
- 1Put a gateway on the partner path
/partner/*through a gateway: validate the key, 50 req/s per key, burst 100,429withRetry-After.yamlroutes: - path: /partner/* auth: api-key quota: { per: api-key, rate: 50/s, burst: 100 } on_limit: { status: 429, headers: { Retry-After: "1" } } upstream: search-partner-poolWhy: Refusing 620 extra requests a second costs a millisecond at the gateway, instead of seconds for guests.
- 2Separate the pools
Partner traffic gets its own Search pool; guests keep theirs.
Why: A bulkhead: even a wrong quota can't starve guest search.
- 3Get the client IP right
Read X-Forwarded-For only for hops our own proxies added, right to left.
Why: Anyone can write that header; forged IPs make IP limits worthless.
- 4Keep the firewall rules tight
Each tier accepts connections only from the tier in front of it. No "temporary" public rule.
Why: The gateway decides how much; the firewall decides who connects at all.
In real life
3 minGitHub's API: quotas per token
- Publicly documents a quota per user or token, with a much lower one for anonymous requests by IP
- Responses show the remaining quota
- Fair to callers sharing an office or cloud IP
Fits APIs where callers identify themselves.
Cloudflare: limits by IP at the edge
- Publicly offers rate-limiting rules, often by IP, before traffic reaches the origin
- Stops floods far from your servers, no keys needed
- Can't tell two users behind one IP apart
Fits anonymous traffic and large floods.
What goes wrong
3 minClients write their own entries and dodge IP limits.
Trust only the rightmost entries added by your own proxies.
IPs are shared; you cut off legitimate traffic with the bad.
Limit by key at a gateway; keep IP blocks for anonymous floods.
Encoded payloads bypass patterns; strict rules block real guests.
Parameterised queries; the WAF is one layer and a temporary patch.
Trade-offs
3 min| Box | Fits | You pay |
|---|---|---|
| L4 load balancer | Non-HTTP, extreme throughput | No path routing, no HTTP retries |
| L7 load balancer / reverse proxy | Most HTTP services | Decrypts and parses everything |
| API gateway | Partner APIs with keys and quotas | Another hop; business logic creeps in |
| WAF | Any public web app, as one layer | False positives; can be bypassed |
- Maya · Product managerPartners bring 22% of bookings. If a big agency gets 429s during a sale, they'll go to a competitor. Just scale Search.
- Omar · On-call SREScaling for a buggy script means paying for servers that answer the same question 600 times a second.
- Maya · Product managerThen at least don't throttle the big ones.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Which box enforces which rule: per-partner quotas at the gateway; "who may connect" in firewall rules.
- The quota (50 req/s per key, burst 100,
429+Retry-After), announced to the 300 partners first. - Which proxies may set
X-Forwarded-For: only ours, read right to left.
A partner's buggy script sent ~670 req/s to /partner/availability and StayHub guest search p95 went from 300 ms to 2.5 s. Plan first; show config changes before applying. Route /partner/* through the API gateway with key validation and a 50 req/s per-key quota, burst 100, answering 429 with Retry-After. Give partners their own Search pool. Read the client IP from X-Forwarded-For only for our proxy range, right to left. Don't add or widen firewall rules. Test: a key over quota, one under, a forged header.
- The trusted proxy range lists only our own proxies, never
0.0.0.0/0. - Over quota gets
429withRetry-After; a forged header doesn't change the logged IP. - Firewall or security group changes in the diff: none.
- It rate-limits by the leftmost
X-Forwarded-Forvalue. - It opens a database port to the internet "just for debugging".
- It silently drops excess partner requests instead of answering
429, triggering their retries.
Check yourself
2 minNext module: Karim gets the first big cloud bill since the Singapore region opened, and nobody can explain €32,000 of it.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You design the partner edge; Lina only checks
Make sure no partner, buggy or not, can slow guest search again, without blocking any partner's legitimate traffic.
- 1List the boxes and what they see
CDN (full request), load balancer (L7, full request), firewall rules (IP and port), services. Nothing in the path knows API keys.
Why: A rule can only live in a box that sees what the rule needs. Per-partner limits need the key.
- 2Put a gateway on the partner path
Route
/partner/*through an API gateway: validate the key, quota 50 req/s per key (burst 100), answer429withRetry-After.Why: Refusing excess at the gateway costs a millisecond; letting it reach Search costs guests seconds.
- 3Separate the pools
Send partner traffic to its own Search pool; guests keep theirs.
Why: A bulkhead (last module): even if a quota is set wrong, partner load can't starve guest search.
- 4Get the client IP right
Configure the app and gateway to read the client IP from X-Forwarded-For only for hops added by our own proxies (right to left).
Why: Anyone can write that header. IP-based limits and logs are worthless if the IP can be forged.
- 5Keep the firewall rules tight
Services accept port 8080 only from the load balancer's group; the database 5432 only from services. Nothing else.
Why: The gateway controls how much; the firewall controls who can connect at all. Lina's check: no "temporary" rule opening anything to the internet.
# nginx.conf (excerpt): L7 routing, per-route timeouts, forwarding headers
upstream search_guests { server 10.0.1.11:8080; server 10.0.1.12:8080; keepalive 64; }
upstream search_partners { server 10.0.3.31:8080; server 10.0.3.32:8080; keepalive 64; }
set_real_ip_from 10.0.0.0/16; # our own proxies only
real_ip_header X-Forwarded-For;
real_ip_recursive on; # walk right to left, skip trusted hops
server {
listen 443 ssl; http2 on;
server_name api.stayhub.com;
location /api/search/ { proxy_pass http://search_guests; proxy_read_timeout 3s; }
location /partner/ { proxy_pass http://search_partners; proxy_read_timeout 5s; }
}Interactive simulation
Visual diagram
Inbound (top) and private/outbound (bottom) paths of a hotel platform. Follow the walkthrough.
Analogy
- In a hotelReception sends requests to free staffIn your networkReverse proxy and load balancer
- In a hotelDoorman checks only which doorIn your networkL4: sees addresses and ports
- In a hotelConcierge reads what you wantIn your networkL7: reads the request
- In a hotelBusiness desk checking badges and quotasIn your networkAPI gateway
- In a hotelKeycard doors on staff corridorsIn your networkFirewall rules
- In a hotelSecurity checking bags at the doorIn your networkA WAF filtering known attacks
Where the analogy breaks: hotel staff can be flexible and use judgement. Network boxes follow their rules exactly, so one wrong rule can block every guest at once.
Check your understanding
Practice ladder
Which box answered?
Debugging Warm-upA partner's POST /partner/bookings goes: CDN → API gateway (timeout 5 s, quota 50 req/s per key) → load balancer → Booking service (takes 8 s today because the payment provider is slow) → database.
For each situation, say which box produces the response and what status code the partner sees:
- The partner sends its 120th request within one second.
- The partner's request reaches Booking, which works for 8 s.
- The partner sends a request with a revoked API key.
- The Booking service's instances are all restarting; the load balancer has no healthy target.
For case 2, also say what happened to the booking itself.
Finish the firewall rules
Design PracticeLina's security groups for the partner path have two gaps. Fill them in so that each tier accepts connections only from the tier in front of it, and explain why referencing groups beats listing IPs.
security_groups:
lb-public:
inbound:
- { port: 443, from: "0.0.0.0/0" }
gateway-tier:
inbound:
- ___
search-partner-pool:
inbound:
- { port: 8080, from: sg:gateway-tier }
db-tier:
inbound:
- ___The gateway listens on port 8443; the database is PostgreSQL; both Search pools read the database.
Access for an analyst and an admin tool
Design PracticeTwo requests land on your desk the same day:
- A data analyst working from home needs read-only access to the production read replica for a week of ad-hoc queries.
- The new internal admin tool (refunds, booking edits) is currently deployed with a public IP "so support can reach it from anywhere", protected by a login page.
Design access for both. Say which boxes you'd use, what each one sees and decides, and what you'd log.
The WAF that blocked Mrs O'Brien
Decision StretchStayHub enabled a managed WAF rule set in block mode last week. Since then, about 0.3% of checkouts are blocked: guests named O'Brien or D'Angelo, addresses like "Rue de l'Église", and some long promo codes trigger the SQL-injection rule. That's about 40 lost bookings a day (of ~12,000).
The booking code already uses parameterised queries everywhere (checked in last month's security review).
Options: - A: switch the SQL-injection rule to log-only for /checkout, keep everything else in block mode. - B: keep it in block mode and add exceptions for the specific fields and patterns seen so far.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
A colleague says: 'We have a firewall, so we don't need a WAF, and we have a load balancer, so we don't need an API gateway.' Explain when they're right and when they're wrong.