The internet, IP addresses and DNS
Thursday, 15:30: StayHub moved to a new server at 10:00, yet 40% of guests still reach the old one, and 72 bookings have landed in the old database since the switch.
How a message crosses the planet in small pieces, and how a website name becomes an address a computer can reach.
- Problem
Billions of devices, owned by thousands of independent networks, need to reach each other by name, without any central switchboard.
- Why the simple system fails
A single central directory or a dedicated wire per conversation can't scale, would be one global point of failure, and would break every time a server moves.
- Engineering concept
Split data into addressed packets that routers forward one hop at a time, and resolve names through a delegated, cached hierarchy.
- Technology
IP (IPv4/IPv6), routers and BGP, DNS resolvers, root/TLD/authoritative nameservers, TTLs.
- Trade-offs
Packets are best-effort (loss, reordering), so reliability lives in TCP. DNS caching makes lookups fast but makes changes slow to propagate.
The problem
2 min- Omar · On-call SRE40% of requests still reach the old server. 72 bookings went into the old database since 10:00.
- You · Engineer at StayHubBut I changed the DNS record myself. It points at the new IP.
- Lina · Senior engineer, your mentorI believe you. Guess where guests get the old address.
The big idea
4 minDNS is a cache with a countdown. The TTL is how long a change takes to reach everyone.
The change was instant at our DNS provider. It reaches a guest only when their resolver's copy expires, and one big resolver serves millions of phones.
How it works
5 minTwo systems work together every time a guest opens StayHub: turns the name into an address, then IP routing delivers the packets.
- 1Ask the resolver
The phone asks its , the server (often its internet provider's) that looks up names and caches answers. A fresh cached answer: about 1 ms.
- 2Walk the hierarchy
On a miss: the root says "ask .com", .com says "ask StayHub's DNS provider", which gives the IP address, the number packets are delivered to. Tens of ms.
- 3Cache it
The resolver keeps the answer for the record's (time to live, in seconds). Ours: 86,400 s, a full day.
- 4Route the packets
The phone sends (small pieces of data with a from- and a to-address). Each router passes them one hop closer, by address only.
- 5Whoever has the IP answers
The old server still has 203.0.113.10 and still works, so it answers, and writes bookings to the old database.
A resolver that looked us up at 09:59 may use the old IP until 09:59 tomorrow. Routers aren't confused: they deliver wherever the phone asked.
Watch a decision being made
4 minLina plans the API's move without a split day
Next month api.stayhub.com moves to a new load balancer. Its TTL is 24 hours. Guests must move within minutes, and no booking may land in the wrong place.
- 1Lower the TTL a full old TTL ahead
Monday 10:00: TTL to 60 s, same address. Wait until Tuesday 10:00, when every 24-hour copy has expired.
Why: Lowering the TTL only affects copies cached after the change. Old copies keep their old countdown.
- 2Make the old target safe
Point the old server at the new database, or make it forward every request to the new address.
Why: Some clients ignore TTLs. If the old target can't do harm, the tail stops mattering.
- 3Switch and verify from outside
Wednesday: change the address. Ask public resolvers, and watch the old server's traffic fall.
bash$ dig @1.1.1.1 api.stayhub.com +short 198.51.100.30 $ dig @8.8.8.8 api.stayhub.com +short 198.51.100.30Why: Your laptop's cache only tells you about you. Other resolvers tell you about guests.
- 4Raise the TTL again
After a week of near-zero old traffic, set the TTL back to 300 s and switch the old server off.
Why: 60 s means more lookups and more dependence on the DNS provider. It's a tool for the move.
In real life
3 minCloudflare: the same IP in many cities (anycast)
- Has publicly described announcing the same IP addresses from data centres worldwide
- Routing itself carries packets to a nearby city
- No DNS countdown when a city goes out of service
Fits a global network that controls its own routing.
Akamai: DNS hands out a nearby address
- Has publicly described its DNS returning a server close to the user's resolver
- Uses short TTLs to steer users again quickly
Fits when DNS answers are computed per user and change often.
What goes wrong
3 minNAT puts a whole office or mobile carrier behind one IP.
Use IPs for rough signals; use sign-in for identity.
Every cached copy lives until its TTL runs out, some longer.
Lower the TTL a full old TTL ahead; verify from several resolvers.
Guests with the old address reach nothing: the site looks down.
Keep it answering or forwarding until its traffic is near zero.
Trade-offs
3 min| TTL | Fits | You pay | Main risk |
|---|---|---|---|
| Long (hours to a day) | Records that never change: mail, nameservers | Slow moves and failovers | A mistake stays cached for hours |
| Short (30 s to 5 min) | Records you might move under pressure | More lookups | A DNS provider outage hits within minutes |
- Karim · CTONever again. Set every DNS record to a 30-second TTL.
- Omar · On-call SREThen resolvers ask our provider every 30 s. If it has an outage, we're unreachable within a minute.
- Karim · CTOAnd with a long TTL, the next emergency move takes a day. We just lived that.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- When to lower the TTL before a move (86,400 s means a full day ahead) and when to raise it again.
- What the old server does after a DNS change: forward to the new one, not keep writing (that split 72 bookings across two databases).
- Who may edit DNS records. A wrong record takes the whole site away, so the agent proposes and you apply.
We're moving www.stayhub.com from 203.0.113.10 to the new load balancer 198.51.100.20. Last time the TTL was 86,400 s and 40% of guests kept reaching the old server for hours, writing 72 bookings to the old database. Write a migration runbook; don't change any DNS yourself. Include: lowering the TTL to 300 s at least 24 hours before, the old server forwarding to the new one, checking what resolvers worldwide return, and the rollback, with verification commands.
- The runbook lowers the TTL at least 24 h before the switch, not at the same moment.
- A guest whose resolver still has the old IP still gets served correctly.
- You run the
digchecks against public resolvers yourself.
- It says DNS changes "propagate in a few minutes" without looking at the TTL.
- It shuts down the old server right after changing the record.
- It edits DNS records or nameservers directly when you asked for a plan.
Check yourself
2 minA record for api.example.com to a new server 20 minutes ago. Your laptop hits the new server, but many customers still hit the old one. The record's TTL was 86,400. What's the most likely explanation?Next episode: one slow report freezes every other request on our server, and Lina opens the box to show what the operating system is juggling inside.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You can watch both systems from a terminal. dig shows a resolver's cached answer and its remaining TTL; traceroute lists the routers on the path.
$ dig www.stayhub.com +noall +answer
www.stayhub.com. 287 IN A 198.51.100.20
# 287 = seconds left before this cached answer expires at your resolver
$ traceroute -n 198.51.100.20
1 192.168.1.1 1.2 ms # home router (NAT)
2 10.20.0.1 8.4 ms # internet provider's network
3 62.115.x.x 11.0 ms # provider backbone
4 198.51.100.20 15.1 ms # StayHub's load balancer, FrankfurtLina plans the API's move without a split day
Next month, api.stayhub.com moves to a new load balancer. Its record has a 24-hour TTL. Plan the switch so guests move within minutes and no booking lands in the wrong place.
- 1Read the current TTL
dig api.stayhub.comat our DNS provider: TTL 86,400 s. That's how long any copy cached right now may live.Why: The longest copy out there decides how early you must start.
- 2Lower the TTL a full old TTL ahead
Monday 10:00: set the TTL to 60 s, same address. Wait until Tuesday 10:00, when every 24-hour copy has expired.
Why: Lowering the TTL only affects copies cached after the change. Old copies keep their old countdown.
- 3Make the old target safe
Point the old server's app at the new database (or make it forward every request to the new address), so a late guest still books into the right place.
Why: Some clients ignore TTLs. If the old target can't do harm, the tail stops mattering.
- 4Switch and verify from outside
Wednesday: change the address. Check with
digagainst a few public resolvers, and watch the old server's request count fall.bash$ dig @1.1.1.1 api.stayhub.com +short 198.51.100.30 $ dig @8.8.8.8 api.stayhub.com +short 198.51.100.30Why: Your own laptop's cache only tells you about you. Other resolvers tell you about guests.
- 5Raise the TTL again
After a week of near-zero old traffic, set the TTL back to 300 s and switch the old server off.
Why: 60 s means more lookups and more dependence on the DNS provider. It's a tool for the move, not a way of life.
| Field / record | What it does | Why you care |
|---|---|---|
| Source / destination IP | Where the packet came from and is going | Firewalls, rate limits and logs use these; NAT rewrites the source |
| IP TTL (hop limit) | Lowered by each router; packet dropped at 0 | Stops packets looping forever (unrelated to DNS TTL) |
DNS A / AAAA | Name → IPv4 / IPv6 address | The record you change when you move a server |
DNS CNAME | Name → another name | How you point a domain at a CDN or load balancer you don't control |
| DNS TTL | Seconds an answer may be cached | Long = fewer lookups, slower changes. Short = agile, more queries |
Visual diagram
First a DNS lookup (top), then packets travel router to router to the server's IP (bottom).
Analogy
- The postal systemA street addressThe internetAn IP address
- The postal systemA long letter sent as numbered postcardsThe internetData split into packets
- The postal systemSorting centres passing mail alongThe internetRouters forwarding hop by hop
- The postal systemA chain of phone booksThe internetDNS: root, then .com, then the site
- The postal system"Valid until" on a copied pageThe internetThe TTL on a cached DNS answer
Where the analogy breaks: postcards take days. Packets cross the planet in a fraction of a second, but they still can't beat the speed of light, so distance always costs some time.
Check your understanding
Practice ladder
Trace three guests after a DNS change
Scenario Warm-upThe record www.stayhub.com had a TTL of 3,600 s (1 hour). At 14:00 we changed it from the old IP to the new one. Three guests use three different resolvers:
- Guest A's resolver last looked us up at 13:50.
- Guest B's resolver last looked us up at 13:10.
- Guest C's resolver had never looked us up before; C opens StayHub at 14:05.
For each guest, say which server they reach at 14:20, and when they'll first reach the new one.
Finish Lina's move checklist
Scenario PracticeStayHub's hotel-owner dashboard, owners.stayhub.com, moves to a new server. Its record has a TTL of 43,200 s (12 hours). The move is planned for Thursday 09:00. Lina's checklist has two gaps:
- Read the current TTL: 43,200 s.
- Lower the TTL: ___ (when, and to what?)
- Make the old target safe: ___
- Switch and verify from outside: change the IP Thursday 09:00; check with
digagainst two public resolvers. - Raise the TTL again after a week of near-zero old traffic.
Fill in steps 2 and 3. Owners edit room rates on this dashboard.
"StayHub is down" in one city only
Debugging PracticeTuesday morning, Maya forwards complaints: guests in Marseille say StayHub doesn't load at all. Guests in Paris and Lyon are fine. Omar sees no errors on our servers and normal traffic.
Yesterday at 18:00 the CDN gave us a new address and someone changed our record. The old CDN address was switched off at 18:30. The record's TTL was 86,400 s.
Explain the most likely cause, how you'd confirm it, and what to do now and next time.
Block the scraper how?
Decision StretchA scraper copies StayHub's hotel prices every night. It uses about 400 different IP addresses, each sending around 60 requests per minute. Normal guests send fewer than 30 per minute. Search is open to guests who aren't signed in.
Options on the table: - A: per-IP limit of 50 requests per minute. - B: block the 400 addresses seen last night. - C: a generous per-IP limit (e.g. 600 per minute) as a backstop, plus detection by behaviour (no images or styles loaded, every hotel in order, all night), with a challenge page for suspicious sessions.
Remember the Lyon office. Write a short decision: "We chose X over Y because…, accepting the risk of…"
Explain it yourself
Explain to a new teammate why a DNS change made an hour ago still sends some users to the old server, and how packets actually reach the new one.