Episode 53 · A database on the open internet

VPCs and subnets: your private network in the cloud

Wednesday, 11:00: an outside security scan finds a copy of StayHub's bookings database answering on a public IP, port 5432 open to the whole internet. Its logs show 41,000 failed logins last week.

Your own private network inside the cloud, where only the front door faces the internet and everything valuable stays out of sight.

30 min intermediateVPCs and public vs private subnetsRoute tables, internet gateways & NATSecurity groups vs network ACLsVPC peering & private connectivityNetwork firewalls vs WAFsLeast privilege for agents

Problem
Your hotel platform's servers and database run in the cloud, and they must talk to each other and to a payment provider, while the database stays unreachable from the internet.
Why the simple system fails
Giving every resource a public IP and relying on passwords exposes databases and admin ports to constant internet scanning; one flat network means one compromised server can reach everything.
Engineering concept
Carve a private address space into subnets per tier and zone. Routes decide what *can* reach the internet; security rules decide what *may* connect. Only the entry point is public.
Technology
VPC, CIDR blocks, public/private subnets, route tables, internet gateway, NAT gateway, security groups, network ACLs, VPC peering, private endpoints, VPN.
Trade-offs
Strong isolation costs complexity and money (NAT gateways, cross-zone traffic, peering limits) and makes debugging connectivity harder. Over-segmenting slows teams; under-segmenting widens the blast radius.

The problem

2 min

Wednesday, 11:00. An outside security scan.

Omar · On-call SRE
52.29.x.x:5432 is open to the internet: our bookings reporting replica. 41,000 failed logins last week.
Omar · On-call SRE
The rule "5432 from 0.0.0.0/0" was added five months ago, "temporarily", for the BI tool.
Karim · CTO
Change the password and delete the rule. Done?

The big idea

4 min

How the database got a public door

The big idea

Two locks, not one: routes decide what can reach a machine; security rules decide what may.

Rules are easy to change, "temporarily". The stronger fix is layout: put the database where there's no road to the internet, so no single rule can expose it.

How it works

5 min

A request through a three-tier network

1
Your own network
A is your private network in the cloud, with an address range you choose (10.0.0.0/16). slice it by tier and zone.
2
Public means "routes to the internet"
There's no public checkbox. A subnet is public only because its sends 0.0.0.0/0 to the internet gateway, the VPC's only front door.
3
Public tier: load balancer only
The only thing facing outside. Its security group accepts 443 from anywhere.
4
Private app tier: out via NAT
No public IPs. A lets them call out (the payment provider) and get replies, but never lets a stranger start a connection in.
5
Data tier: no road out at all
No route to the internet, not even NAT. Unreachable from outside, whatever any rule says.
6
Security groups: the second lock
A guards each machine and remembers connections. sg-app accepts 8080 only from sg-lb; sg-db accepts 5432 only from sg-app. A network ACL guards a whole subnet and doesn't remember.

Watch a decision being made

4 min

Worked example

Lina rebuilds Frankfurt's network

Only the load balancer public, the database unreachable from the internet, and the BI tool still working.

1Plan the address space
Frankfurt 10.0.0.0/16, Singapore 10.1.0.0/16, staging 10.10.0.0/16: never overlapping, in one shared plan.
Why: Linking networks later needs distinct ranges; overlaps mean renumbering.
2Cut subnets by tier and zone
Per zone (3): public /24, app /20 (4,096 addresses), data /24.
Why: Containers can take one address each, so app subnets need room. One subnet per zone keeps a tier alive if a zone fails.
3Write the routes
Public → internet gateway. App → the NAT gateway in the same zone. Data: no default route.
hcl
```
# private app subnets (zone A)
route { cidr_block = "0.0.0.0/0"
        nat_gateway_id = aws_nat_gateway.nat_a.id }
# data subnets: no 0.0.0.0/0 route at all
```
Why: The first lock. A NAT per zone means one zone failing doesn't cut the others off.
4Rules by group, and a proper door for BI
Security groups reference each other, not IPs. The BI vendor's connector runs in the app tier and dials out; delete the replica's public IP.
Why: Groups stay correct as machines come and go. The outside world never connects in.

In real life

3 min

Trust the network versus trust no network

The classic perimeter: private network + VPN

The pattern in every major cloud's reference designs: private subnets, VPN or bastion for people
Inside, systems mostly trust each other
Simple; a breach inside can spread

Fits most companies as the baseline.

Google BeyondCorp: zero trust

Google has publicly described not trusting its internal network for employee access
Every request checks user and device identity
Needs strong identity and device management

Fits large organisations that can invest in identity everywhere.

What goes wrong

3 min

A public database "with a strong password"

It's scanned constantly; any leaked credential is exploitable.

Instead

Data subnets with no internet route; only the app tier's group may connect.

Opening a rule to 0.0.0.0/0 "temporarily"

Temporary rules are forgotten and stay.

Instead

The narrowest rule, through code review, with an expiry; debug with flow logs.

Overlapping address ranges

You find out the day you need to connect two networks.

Instead

Allocate every range from one central plan.

Trade-offs

3 min

Layout	Fits	You pay
Flat network (default VPC)	Throwaway prototypes	Every machine one rule from exposure
Three-tier private network	Any production system: the default	NAT costs; harder debugging
Several VPCs, peered	Teams, compliance scopes like payments	IP planning and routing as a discipline

Friday, 10:00. Where should payment data live?

Lina · Senior engineer, your mentor
Payment services in their own VPC. The card-data audit then covers only that small network.
Karim · CTO
Another VPC means peering, more routes, two networks to debug at 3 a.m. Separate subnets are enough.
Lina · Senior engineer, your mentor
In one VPC, every service is one rule away from payments. The audit counts all of them.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The address plan, never overlapping. An agent picking "any free range" can block peering for years.
What is public: only the load balancers. Never a database, whatever is convenient this week.
How outside tools get in: connectors that dial out, not open database ports.

Tell Claude Code

In our infrastructure code, add the data tier for the Frankfurt VPC (10.0.0.0/16): private subnets 10.0.64.0/24, 10.0.65.0/24, 10.0.66.0/24, one per zone, with only the local route. Move the bookings reporting replica there; its security group allows 5432 only from the app tier's group. No 0.0.0.0/0 rule; don't touch Singapore. Don't apply: show the plan output, every changed rule, and how you'll prove the replica is unreachable from outside but reachable from app servers.

Check before you accept

No 0.0.0.0/0 in a security group, and no internet-gateway route on a private subnet.
A connection test from an app server succeeds; one from outside times out.
Rules reference security groups, not IP lists.

Red flags

It opens a port to 0.0.0.0/0 "to unblock the test".
It gives a database a public IP so a tool can connect.
It fixes an outbound timeout by making a subnet public instead of adding a NAT route.

Check yourself

2 min

Q1/3Reasoning

What makes a subnet 'public'?

How confident are you?

Next episode: on a Thursday afternoon, one of the three Frankfurt zones loses power, and you find out which parts of StayHub really live in all three.

Finished?

Mark it complete to update your skill map.

Next: Regions and availability zones

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Routes and rules, step by step

Worked example

Lina rebuilds Frankfurt's network

Move Frankfurt out of the default network into a VPC where only the load balancer is public, the database can't reach or be reached from the internet, and the BI tool still works.

1Plan the address space
Prod Frankfurt 10.0.0.0/16, prod Singapore 10.1.0.0/16, staging 10.10.0.0/16. Never overlapping, written in one shared plan.
Why: Linking networks later (peering, VPN) needs distinct ranges. Overlaps are discovered the day you need to connect, and fixing them means renumbering.
2Cut subnets by tier and zone
Per zone (3 zones): public /24 (10.0.0.0–10.0.2.0), app /20 (10.0.16.0, .32.0, .48.0), data /24 (10.0.64.0–10.0.66.0).
Why: App subnets get room to grow (4,096 addresses each), because containers can take one address each. One subnet per zone per tier keeps each tier alive if a zone fails.
3Write the routes
Public: 0.0.0.0/0 → internet gateway. App: 0.0.0.0/0 → the NAT gateway in the same zone. Data: no default route.
Why: Routes are the first lock. A NAT per zone means losing one zone doesn't cut the others off from the payment provider.
4Write the security groups
sg-lb: 443 from anywhere. sg-app: 8080 from sg-lb. sg-db: 5432 from sg-app and from sg-bi-connector.
Why: The second lock. Referencing groups instead of IPs stays correct as machines come and go.
5Give the BI tool a proper door
Run the BI vendor's connector in the app tier; it dials out through NAT to the BI service and reads the replica over 5432. Delete the replica's public IP.
Why: The outside world never connects in. The original "temporary" rule has no reason to exist.

	Security group	Network ACL
Applies to	Instances / network interfaces	A whole subnet
State	Stateful: replies to allowed traffic are allowed	Stateless: return traffic needs its own rule
Rules	Allow only	Allow and deny, in number order
Can reference	Other security groups	Only address ranges
Typical use	Main, fine-grained control per tier	Coarse guardrails ("data subnets never talk to the internet")

Visual diagram

Internet

Internet gateway

Load balancer (public subnets)

NAT gateway (public subnet)

App, zone A (private)

App, zone B (private)

DB primary (data subnet)

DB standby, zone B (planned)

Analytics VPC (peered)

Click a component

StayHub's Frankfurt production VPC (10.0.0.0/16), three tiers, shown with two of its three zones. Start the walkthrough.

Analogy

An office buildingA cloud network

An office buildingThe building with its own room numbersA cloud networkA VPC with its address range
An office buildingThe lobby floor with street doorsA cloud networkA public subnet
An office buildingStaff floors and the vaultA cloud networkPrivate subnets
An office buildingCorridor signs to each staircaseA cloud networkRoute tables
An office buildingThe staff mailroom: out, never inA cloud networkA NAT gateway
An office buildingKeycard readers on each doorA cloud networkSecurity groups

Where the analogy breaks: a building can be rebuilt slowly. A VPC's address range is very hard to change once things are running, so pick it carefully on day one.

Check your understanding

Q1/3Reasoning

How many IP addresses does a /20 subnet contain (before the cloud provider reserves a few)?

addresses

How confident are you?

Practice ladder

Step 1 · Trace

Will this connection work?

Debugging Warm-up

In Lina's new Frankfurt VPC (public, app and data subnets; NAT per zone; sg-lb, sg-app, sg-db as designed), trace each connection: does it succeed? If not, which lock stops it, the route or the security group?

A scanner on the internet tries 10.0.64.12:5432 (the database's private IP).
An app server calls the payment provider's API on 443.
The load balancer forwards a request to an app server on 8080.
A compromised app server tries to download a tool from the internet into the data subnet's database host (a process on the DB host starts the download).
The BI connector (sg-bi-connector, app tier) queries the replica on 5432.

Step 2 · Complete

Finish the address plan

Design Practice

Lina's address plan has gaps. Fill them in, keeping her pattern, and answer the two questions.

Network	Range
Prod Frankfurt	10.0.0.0/16
Prod Singapore	10.1.0.0/16
Staging	___
Office VPN clients	10.200.0.0/22

Frankfurt subnets:

Tier	Zone A	Zone B	Zone C
Public /24	10.0.0.0/24	10.0.1.0/24	10.0.2.0/24
App /20	10.0.16.0/20	10.0.32.0/20	___
Data /24	10.0.64.0/24	___	10.0.66.0/24

About how many addresses does each /20 hold?
Why must staging not reuse 10.0.0.0/16, even though it's a separate VPC?

Step 3 · Solve

Design Singapore's network

Design Practice

Singapore was built quickly last month. Design it properly in 10.1.0.0/16, three zones:

An internet-facing load balancer.
4–30 autoscaled API VMs (one address each; containers later may need one address each too).
Managed PostgreSQL (primary + standby) and a Redis cache.
API VMs call the payment provider and upload photos to object storage (about 1 TB a day).
Engineers sometimes need a database shell.

Give subnets per tier and zone, routes, security groups, and how engineers and object storage are reached.

Step 4 · Decide

The acquired startup with the same addresses

Decision Stretch

StayHub buys a small tours startup. Its VPC also uses 10.0.0.0/16, the same as prod Frankfurt. StayHub's booking service must call the tours API (about 50 req/s) within 3 months.

Option A: renumber the startup's VPC into 10.20.0.0/16 (rebuild subnets, move ~15 services), then peer the VPCs.
Option B: don't connect the networks; expose the tours API to StayHub through a private service endpoint (PrivateLink-style), which works even with overlapping ranges, as a one-way door for that one API.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain how a guest's request reaches a database in a private subnet, and why an attacker on the internet can't connect to that database directly even if they know its IP and password.

0 words