HTTP and TLS: the language of the web, encrypted
Saturday, 00:00 UTC: every browser starts refusing api.stayhub.com with "Your connection is not private", and bookings drop from 200 an hour to zero. Nothing was deployed. Nothing crashed.
What your browser and your server actually say to each other, and how the padlock keeps strangers out of the conversation.
- Problem
A hotel search page needs 80 resources from a server 100 ms away, privately, without anyone on the café Wi-Fi reading card numbers or swapping prices.
- Why the simple system fails
Plain HTTP/1.1 sends readable text, handles one request at a time per connection, and a fresh connection per request spends most of the time on handshakes instead of data.
- Engineering concept
Reuse connections, multiplex many requests over them, let caches skip requests entirely, and wrap everything in TLS so the server proves its identity and traffic is encrypted.
- Technology
HTTP/1.1, HTTP/2, HTTP/3 over QUIC, Cache-Control and ETag, keep-alive, TLS 1.3, X.509 certificates and certificate authorities.
- Trade-offs
Every round trip saved adds complexity (multiplexing, UDP-based transport, cache invalidation). TLS costs a round trip and certificate operations; skipping it is never acceptable on the public internet.
The problem
2 min- Omar · On-call SRE100% of API calls failing since 00:00:00. Browsers: "Your connection is not private". Bookings 200/hour → 0.
- Karim · CTOWho deployed on a Friday night?
- Omar · On-call SRENobody. Servers healthy, CPU 12%.
The big idea
4 minEvery certificate has an expiry date. Renew automatically, and watch every one.
The certificate was bought by hand and renewed once by an engineer who has since left. The fix took until 01:50, because someone had to find the CA login: about 370 bookings lost. The CDN's certificates renew themselves; they were fine.
How it works
5 min- 1HTTP inside TLS
is the web's request/response language: method, path, headers, a status code. encrypts the connection and proves the server's identity. Together: HTTPS.
- 2Client hello
In the , the browser sends the name it wants (: api.stayhub.com), the protocols it speaks (h2, h3) and a key share.
- 3The server shows its ID
A is the server's ID card: name, public key, validity dates, and a signature from a certificate authority (CA) your device already trusts. The server sends it plus the intermediate that signed it.
- 4The browser checks, locally
Name matches? Dates valid? Chain of signatures leads to a trusted root? No call to the CA, no grace period. Any check fails: it hangs up.
- 5Encrypted HTTP
Both sides derive the same keys; the request goes out encrypted. TLS 1.3 adds one round trip on top of TCP's.
- 6Headers decide who keeps a copy
Cache-Controlsays who may store a response: hashed files a year, hotel datamax-age=60with an (unchanged → a tiny304), anything personalprivate, no-store.
Watch a decision being made
4 minLina makes sure it never happens again
A hand-bought certificate cost 1 h 50 min of downtime. Make every certificate renew itself, and know weeks before any could expire.
- 1Find every certificate
List every hostname and port serving TLS, public and internal, and read each end date.
bashopenssl s_client -connect api.stayhub.com:443 -servername api.stayhub.com </dev/null \ | openssl x509 -noout -subject -issuer -enddateWhy: You can't renew what you don't know exists. Internal certificates are the forgotten ones.
- 2Check the whole chain
With
-showcerts, confirm each server sends its certificate and the intermediate.Why: Browsers fill in a missing intermediate; servers and phones often don't.
- 3Automate renewal
ACME on the load balancer, CDN-managed certificates elsewhere, renewing at two-thirds of the lifetime.
Why: A bot that renews daily can't be forgotten when someone leaves.
- 4Alert on days left
A daily external check alerts at 30, 14 and 7 days; at 7 it pages on-call. Prove it in staging with a 2-day certificate.
Why: Automation fails silently too (expired DNS token, firewall change). Watch the outcome, not the process.
In real life
3 minLet's Encrypt: 90 days, automated
- Publicly offers free 90-day certificates through the ACME protocol
- The short lifetime is meant to push everyone to automate
- A renewal bot runs daily; humans never touch it
Fits any service where a bot can prove control of the domain.
A paid certificate, installed by hand
- Commercial CAs sell certificates valid up to about a year, the maximum browsers accept
- Bought on a card, installed manually
- Renewal depends on a calendar reminder and a login
Fits rare cases needing a specific CA, and only with monitoring.
What goes wrong
3 minIt accepts any impostor, often for every call in the process.
Fix the chain; trust a specific intermediate only for the client that needs it.
Someone forgets or leaves; at expiry every client fails at once.
ACME automation, an inventory, alerts well before expiry.
A CDN serves the first guest's bookings to everyone.
Default per-user responses to private, no-store.
Trade-offs
3 min| Option | Fits when | You pay |
|---|---|---|
| TLS ends at the load balancer | Small systems, a temporary stage | Internal traffic readable by anyone inside |
| Re-encrypt to every service (TLS or mTLS) | Payments, personal data, compliance | Many certificates to automate; harder debugging |
- Lina · Senior engineer, your mentorNext: encrypt inside too. Today it's plain HTTP behind the load balancer.
- Karim · CTOWe just lost two hours to one certificate. You want a hundred more?
- Lina · Senior engineer, your mentorOur network is private until one machine inside is compromised. Then every token between services is readable.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The
Cache-Controlrule per kind of response. Claude copies whatever header the nearest file uses. - That certificate verification is never turned off, anywhere. Broken chains get fixed.
- How certificates renew and when you're warned: 30, 14 and 7 days, paging at 7, public and internal.
api.stayhub.com was down 1 h 50 min because a hand-bought certificate expired; about 370 bookings lost. Don't change production yet. List every hostname and port in our infrastructure config that serves TLS, public and internal, with the command to read its expiry and check the full chain. Then propose ACME renewal on the load balancer at two-thirds of the lifetime and a daily check alerting at 30, 14 and 7 days. Never disable verification. Include a staging test with a 2-day certificate.
- Compare Claude's hostname list with what you know exists. A missing one is a future outage.
- Search the diff for
NODE_TLS_REJECT_UNAUTHORIZEDandrejectUnauthorized: false: zero hits. - Block renewal in staging and see the alert fire.
- It "fixes" a chain error by disabling verification.
- It adds
Cache-Control: publicto a response that depends on who is logged in. - It skips the expiry alert because "renewal is automated".
Check yourself
2 minNext episode: StayHub moves its API to a new load balancer, changes one DNS record, and hours later some guests still land on the old one.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
Lina makes sure it never happens again
api.stayhub.com went down for 1 h 50 min because a hand-bought certificate expired. Make every StayHub certificate renew itself, and make sure we'd know weeks before any of them could expire.
- 1Find every certificate
List every hostname and port that serves TLS, public and internal: api, www, partners, the admin panel, service-to-service endpoints. For each, read the end date with
openssl s_client … | openssl x509 -noout -enddate.Why: You can't renew what you don't know exists. Internal certificates are the ones most often forgotten.
- 2Check the whole chain
Run
openssl s_client -showcertsand confirm each server sends its certificate and the intermediate. Fix any that send only their own.Why: Browsers often fill in a missing intermediate themselves; servers and phones often don't. A chain that "works in Chrome" can still break the app.
- 3Automate renewal
Use ACME on the load balancer (and the CDN's managed certificates where available), renewing at two-thirds of the lifetime.
Why: A renewal that happens daily, by a bot, can't be forgotten when someone leaves. Renewing early leaves weeks to fix a failing renewal.
- 4Alert like it's disk space
An external check reads every certificate's end date daily and alerts at 30, 14 and 7 days; at 7 days it pages on-call.
Why: Automation fails silently too (expired DNS token, changed firewall). The alert catches the failure while there's still time.
- 5Prove it
In staging, issue a certificate valid for 2 days and watch the bot renew it and the alert fire if you block the renewal.
Why: A renewal path that has never run is a hope, not a process.
HTTP methods carry meaning that caches and retry logic rely on. GET and HEAD are safe and cacheable. GET, PUT and DELETE are idempotent. POST is neither, which is why payment endpoints need . Responses say who may keep a copy with Cache-Control, and an lets a cache ask "has it changed?" and get a tiny 304:
GET /api/hotels/42 HTTP/1.1
Host: api.stayhub.com
HTTP/1.1 200 OK
Cache-Control: public, max-age=60, stale-while-revalidate=300
ETag: "h42-v918"
Vary: Accept-Language
Strict-Transport-Security: max-age=31536000; includeSubDomains
{"id":42,"name":"Hotel Ribeira","stars":4, ...}
# 90 s later the copy is stale, so the browser revalidates:
GET /api/hotels/42 HTTP/1.1
If-None-Match: "h42-v918"
HTTP/1.1 304 Not Modified| Cache-Control directive | Meaning | Typical use |
|---|---|---|
| max-age=N | Fresh for N seconds; no request needed | Photos, versioned JS (long N) |
| s-maxage=N | Freshness for shared caches (CDNs) only | Public pages cached 30 s at the edge |
| public / private | May shared caches store it? private = browser only | private for anything per-user |
| no-cache | Store it, but revalidate before every use | HTML that must never be stale |
| no-store | Never store anywhere | Bookings, payment pages, personal data |
| immutable | Never revalidate while fresh | Files whose URL contains a content hash |
| stale-while-revalidate=N | Serve stale for N s while refreshing | Search results, hotel lists |
Interactive simulation
Visual diagram
One HTTPS request to a hotel API in a common set-up where TLS ends at a CDN edge (StayHub's api.stayhub.com ends TLS at its own load balancer in Frankfurt). Start the walkthrough.
Analogy
- Calling a hotel deskOne question per call, six phone linesHTTP and TLSHTTP/1.1
- Calling a hotel deskTen questions on one line, answers numberedHTTP and TLSHTTP/2 multiplexing
- Calling a hotel deskNot hanging up between questionsHTTP and TLSKeep-alive, reusing connections
- Calling a hotel deskClerk shows an ID signed by the governmentHTTP and TLSA certificate signed by an authority
- Calling a hotel deskAgreeing on a secret codeHTTP and TLSTLS session keys
- Calling a hotel desk"This price list is valid for an hour"HTTP and TLSA Cache-Control header
Where the analogy breaks: a clerk can glance at an ID and use judgement. Your browser checks the certificate with strict maths, and refuses outright if anything is off, including a date one minute past expiry.
Check your understanding
Practice ladder
Trace three handshakes
Debugging Warm-upThree StayHub hostnames, three certificate situations. For each, say whether a browser connects, whether the mobile app's HTTP client connects, and why.
| Host | Certificate served | Dates | Chain sent |
|---|---|---|---|
| api.stayhub.com | CN = api.stayhub.com | valid | leaf + intermediate |
| partners.stayhub.com | CN = api.stayhub.com | valid | leaf + intermediate |
| img.stayhub.com | CN = img.stayhub.com | valid | leaf only |
Finish Lina's certificate runbook
Design PracticeLina's runbook for "a certificate alert fired: 7 days left on partners.stayhub.com" has two blank steps. Fill in steps 3 and 5.
- Confirm the alert:
openssl s_client -connect partners.stayhub.com:443 -servername partners.stayhub.com | openssl x509 -noout -enddate. - Check the renewal bot's log for that hostname.
- ___
- Fix the cause (for example a renewed DNS API token, or the hostname added to the bot's list), then trigger a renewal.
- ___
- Write down what broke in the postmortem doc, and add a check so this cause would alert sooner.
Make the hotel page fast on a second visit
Design PracticeEvery resource on the hotel page is served today with:
Cache-Control: no-store Connection: close
Guests in Spain see 1.8 s loads every time, even on repeat visits. The page has: the HTML (per-guest member prices), /static/app.3f9a.js and /static/app.7c21.css (content hash in the name), 12 hotel photos (change a few times a year), /api/hotels/42 (description, changes a few times a month) and /api/hotels/42/availability (changes every minute, same for everyone).
Propose headers for each, and explain what changes for a second visit.
HSTS preload for stayhub.com?
Decision StretchLina proposes adding stayhub.com to the browsers' HSTS preload list: browsers would then refuse to connect to stayhub.com or any subdomain over plain HTTP, forever, even on a first visit. Removal takes months.
Facts: every guest-facing host is already HTTPS. But legacy-extranet.stayhub.com, used by 40 small hotels, still runs on plain HTTP on an old server, and a vendor's printer integration at some hotels posts to print.stayhub.com over HTTP.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Explain to a junior colleague what happens between fetch('https://api.stayhub.com/hotels/42') and the JSON arriving, and what you would change to make the second call faster.