DNS deep dive: records, TTLs, GeoDNS and failover
Wednesday, 14:00: StayHub points api.stayhub.com at a new load balancer and switches the old one off an hour later. At 20:00, 30% of guests are still trying the old address, and failing.
DNS does more than turn names into numbers. It routes mail, steers visitors to the nearest server, and causes a surprising number of outages.
- Problem
Your hotel platform runs in Europe and Asia, sends booking emails, uses a CDN and a dozen SaaS tools, and must keep working when a region goes down.
- Why the simple system fails
A single A record pointing at one IP can't send users to the nearest region, can't route around a dead one, can't express 'mail goes elsewhere', and every change is held hostage by caches you don't control.
- Engineering concept
Use the right record type for each job, choose TTLs deliberately, and let health-checked, location-aware authoritative DNS steer users, while remembering that every answer may be cached longer than you asked.
- Technology
A, AAAA, CNAME, ALIAS, MX, TXT, NS, CAA records; recursive resolvers; TTL and negative caching; GeoDNS, latency-based routing, anycast, health-checked DNS failover.
- Trade-offs
Low TTLs mean fast changes but more queries and more dependence on DNS availability. DNS steering is cheap and global but coarse and slow compared to a load balancer.
The problem
2 min- Omar · On-call SREAt 14:00 we pointed api.stayhub.com at the new load balancer. At 15:00 we switched the old one off.
- Omar · On-call SREAt 20:00, 30% of guests still try the old address and fail. Our DNS provider shows the new one.
- You · Engineer at StayHubThe record is right. So where does the old answer come from?
The big idea
4 minA DNS change is a request to caches you don't control. Lower the TTL first, and keep the old path alive until they agree.
My laptop showed the new IP only because its cache happened to expire. That's why DNS bugs look random: everyone's answer depends on when their cache last asked.
How it works
5 min- 1Records, each with a TTL
A record is one entry for a name: A (an IPv4 address), CNAME (another name), MX (mail), TXT (proofs), NS (who answers). Its says how many seconds anyone may keep a copy.
- 2The phone asks its resolver
A resolver looks names up for a device, usually run by its internet provider, or a public one like 1.1.1.1. It caches answers.
- 3Cache hit: answer at once
If the resolver's copy is younger than its TTL, it answers from it. Our provider is never asked, so our change is invisible.
- 4Cache miss: walk the chain
Root → .com → our provider's nameservers, which give the current record. The resolver keeps it for the TTL.
- 5Steering with answers
answers Jakarta with Singapore's address and Paris with Frankfurt's; drops a region that fails health checks. Both are bounded by the same caches.
Watch a decision being made
4 minYou plan the next move; Lina checks it
Next month, api.stayhub.com moves again. Plan it so no guest sees an error, whatever their resolver cached.
- 1Read the current TTL from the source
Ask the authoritative server, not a cache: TTL 86,400 s again.
bashdig +norecurse api.stayhub.com A @ns1.dns-provider.example # the truth dig api.stayhub.com A @1.1.1.1 # what guests seeWhy: The old TTL decides how early you must start.
- 2Lower it one old TTL ahead
Day −1, 14:00: TTL 60 s, same address. Wait at least 24 h.
Why: Resolvers holding the 24-hour copy must expire it and pick up the short TTL first.
- 3Change, verify, keep the old path
Day 0: new IP. Check at our provider, two public resolvers, a phone on mobile data. Switch the old load balancer off only after an hour near zero traffic.
Why: Comparing authoritative and resolver answers separates "wrong record" from "stale cache".
- 4Restore a sensible TTL
Raise it to 300 s; write it in the runbook.
Why: 60 s forever means more lookups and more dependence on the DNS provider every minute.
In real life
3 minAWS Route 53: steering with DNS answers
- Publicly documents latency, geolocation and failover routing with health checks
- Each region keeps its own IP; DNS picks one per resolver
- Failover speed = health checks + TTL
Fits picking a region per user, cheaply.
Cloudflare: anycast
- Publicly describes announcing the same IPs from data centres worldwide
- Internet routing sends each user to a nearby location
- A location is withdrawn in seconds, no DNS cache to wait for
Fits fast global failover, bought as a CDN or global load balancer.
What goes wrong
3 minResolvers keep the old value until their copy expires.
Lower the TTL one full old TTL ahead.
Caches still send users there.
Switch off by traffic, not by the clock.
Connection pools keep an IP for the life of the process.
Respect TTLs, cap connection lifetime (say 5 minutes), re-resolve after repeated connect errors.
Trade-offs
3 min| Option | Fits | You pay |
|---|---|---|
| Short TTLs (30–60 s) | Failover endpoints; records about to change | More lookups; a DNS outage hits sooner |
| Long TTLs (hours) | NS, MX, TXT, stable records | Mistakes take hours to fix |
| Anycast / global load balancer | Failover in seconds | A paid provider or BGP skills |
- Karim · CTOSimple fix: 30-second TTLs on everything. Then we can change anything, any time.
- Omar · On-call SREThen a 10-minute DNS provider outage makes us unreachable for 10 minutes. With an hour, most guests wouldn't notice.
- Karim · CTOHow often does a DNS provider go down?
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- The migration timeline: TTL 86,400 → 60 s at least 24 h before the change, 300 s afterwards.
- When the old load balancer may go: after an hour near zero traffic, never on a schedule.
- What DNS is for: coarse, slow steering. A 5% canary belongs at the load balancer.
api.stayhub.com moves next month. Last time we changed the record with a 24-hour TTL and switched the old load balancer off after an hour; about 30% of requests failed until the next day. Don't touch DNS yet. Write docs/runbooks/api-dns-move.md: read the TTL from our authoritative server with dig, lower it to 60 s at least 24 hours ahead, change, verify at our provider, two public resolvers and mobile data, keep the old load balancer until traffic is near zero for an hour, then TTL 300 s. Give commands, expected output and rollback per step.
- The TTL lowering comes at least one old TTL (24 h) before the address change.
- Verification queries the authoritative server and a public resolver separately.
- The old load balancer's switch-off condition is a traffic number, not a time.
- It lowers the TTL and changes the address in the same edit.
- It says "wait 24–48 hours for propagation" instead of reading the TTL.
- It proposes weighted DNS for a 5% canary that must roll back in 30 seconds.
Check yourself
2 minNext episode: StayHub's partner API gets 40,000 requests a minute from one IP address, and you lead the response.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You plan the next move; Lina checks it
Next month, api.stayhub.com moves again, this time to a load balancer in a new network. Plan it so no guest sees an error, whatever their resolver cached.
- 1Read the current TTL
dig +norecurse api.stayhub.com @ns1.dns-provider.example→ TTL 86,400 s again (someone reset it after the last move).Why: The old TTL decides how early you must start. Lina's check: always read it from the authoritative server, not a cache.
- 2Lower it one old TTL ahead
Day −1, 14:00: set the TTL to 60 s, without changing the address. Wait at least 24 h.
Why: Resolvers holding the 24-hour copy must expire it and pick up the new, short TTL. Only then can a change spread in about a minute.
- 3Make the change and verify
Day 0, 14:00: point the record at the new IP. Check with
digat our provider and at two public resolvers, and from a phone on mobile data.Why: Comparing the authoritative answer with resolver answers separates "wrong record" from "stale cache".
- 4Keep the old path alive
Leave the old load balancer running. Watch its request rate; switch it off only when it's near zero for an hour (expect a small tail from clients that ignore TTLs).
Why: Wednesday's outage came from switching off a target that caches still pointed to.
- 5Restore a sensible TTL
Raise the TTL to 300 s and note it in the runbook, with the reason.
Why: 60 s forever means more lookups and more dependence on the DNS provider being up every minute. 300 s is a fair default for an entry point.
| TTL | Change spreads in | Query load | Good for |
|---|---|---|---|
| 30–60 s | About a minute (plus misbehaving caches) | Many cache misses; each adds a lookup | Failover-managed endpoints; records you're about to change |
| 300 s | Minutes | Moderate | Default for app and CDN entry points |
| 3,600 s | Up to an hour | Low | MX, TXT, rarely changed records |
| 86,400 s | Up to a day | Minimal | NS and truly static records |
| Steering technique | How it decides | Caveat |
|---|---|---|
| GeoDNS | Location of the resolver (or client subnet, if the resolver sends it) | A guest using a far-away public resolver can be sent to the wrong region |
| Latency-based routing | Measured latency from the resolver's network to each region | Same resolver-location caveat |
| Weighted records | Fixed percentages, e.g. 90/10 | Not exact because of caching |
| Failover records | Primary while healthy, else secondary | Recovery = detection + TTL + client caching |
| Anycast | Internet routing to the nearest location announcing the same IP | Not DNS logic at all; needs a provider or BGP skills |
Interactive simulation
Visual diagram
GeoDNS with health-checked failover across two regions.
Analogy
- A hotel's address"Hotel Central is at 12 Main Street"DNSAn A record
- A hotel's address"Ask for Main Street Hotels Group"DNSA CNAME pointing to another name
- A hotel's address"Post goes to the station PO box"DNSAn MX record for email
- A hotel's addressAddress cards travel agents keep for a yearDNSA long TTL, cached elsewhere
- A hotel's addressAsian callers routed to the Asian centreDNSGeoDNS
Where the analogy breaks: travel agents might call to check. DNS caches never ask; they trust their copy until the TTL is up, and a few keep it even longer.
Check your understanding
Practice ladder
Trace a Singapore failover
Scenario Warm-upapi.stayhub.com uses latency routing with failover: health checks every 10 s, unhealthy after 3 failures, TTL 60 s. Singapore dies at 03:00:00.
Trace when each of these guests stops being sent to Singapore:
- Guest A's resolver cached the Singapore answer at 02:59:50.
- Guest B's resolver cached it at 03:00:25.
- Guest C's app resolved it at 02:58:00 and keeps an HTTP/2 connection open for as long as it works.
- Guest D's resolver has no cached copy.
Fill in Lina's migration runbook
Design PracticeLina's runbook for moving partners.stayhub.com (TTL 3,600 s) to a new load balancer has two blank steps.
- Read the current TTL from the authoritative server.
- ___
- Change the A record to the new IP.
- Verify: authoritative server, 1.1.1.1, 8.8.8.8, and a phone on mobile data.
- ___
- Raise the TTL to 300 s and update the runbook.
Fill in steps 2 and 5 with times and the reason for each.
Move DNS providers without downtime
Design Practicestayhub.com is hosted on the registrar's basic DNS. Its NS records have a TTL of 172,800 s (2 days). You want to move the zone to a managed DNS provider with health checks and latency routing.
The zone has about 40 records (A, CNAME, MX, TXT for email authentication, CAA). Booking emails and certificate renewals must keep working throughout.
Plan the move so that no guest, email or certificate renewal fails at any point.
Canary the new search API with DNS?
Decision StretchThe search team wants to send 5% of traffic to a new search API version and roll back within 30 seconds if errors rise. They propose weighted DNS records: search.stayhub.com → 95% old IP, 5% new IP, TTL 60 s.
The alternative: a single DNS name for the load balancer, and a 95/5 weighted rule on the load balancer that can be changed instantly.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Your Asia region just went down. Explain to your product manager why some Asian guests recovered in one minute, some in four, and a few only after reinstalling the app.