Episode 63 · The photos that vanished

Volumes and networks: where container data lives and how containers talk

Monday, 9:10: 37 hotels uploaded 212 new photos over the weekend, and after this morning's redeploy every one of them returns 404. The container restarted cleanly. The photos didn't come back.

Containers forget everything when they're replaced. Here's where your data should live instead, and how containers find each other.

30 min intermediateBind mounts vs named volumesService discovery by name on user-defined networksPort publishing and host exposureVolumes & container networksStateless servers

Problem
The booking API runs in a container next to PostgreSQL and Redis. After a redeploy, uploaded hotel photos vanish; the API can't reach the database at `localhost:5432`; and someone notices Redis is reachable from the internet.
Why the simple system fails
Data written inside a container's writable layer dies with it; `localhost` inside a container means the container itself; and publishing every port to 0.0.0.0 exposes internal services.
Engineering concept
Keep containers disposable and put state in explicit storage. Let containers talk over a private network by name, and publish to the host only the ports users need.
Technology
Named volumes, bind mounts, tmpfs, user-defined bridge networks, Docker's embedded DNS, port publishing (-p), object storage and managed databases in production.
Trade-offs
Explicit storage and networks add configuration and choices (performance, permissions, backups), but make containers replaceable, scalable and safer.

The problem

2 min

Monday, 9:10. The owner portal's first weekend in containers.

Maya · Product manager
37 hotels uploaded 212 photos over the weekend. Every one returns 404 since this morning.
Omar · On-call SRE
owner-api was redeployed at 08:50. Container healthy, disk not full.
Lina · Senior engineer, your mentor
You lead the first guess. Where did those files go?

The big idea

4 min

Where did 212 photos go?

The big idea

Containers are disposable. Data must live somewhere explicit, outside them.

A container is a hotel room, emptied after every guest; a volume is the luggage room downstairs. Containers also have their own network, which is the next thing you'll trip on.

How it works

5 min

Where files live, and how containers find each other

1
Writable layer
On top of the read-only image layers, each container gets a thin writable layer. Writes to /app/uploads land there, and it's deleted with the container.
2
Named volume
A is storage Docker manages on the host: -v uploads:/app/uploads. It survives replacement, but lives on one host's disk and is not a backup.
3
Bind mount
A bind mount maps a host folder into a container: great for live code in development, fragile in production.
4
Off the host
Object storage keeps files outside the VM, so any server can read them.
5
localhost is the container itself
Each container has its own network stack. On a user-defined network (docker network create owners-net), containers reach each other by name: db:5432.
6
Publish only the front door
-p 80:3000 opens a port on every interface of the VM. Containers on the same network talk without it; publish only what users need.

Watch a decision being made

4 min

Worked example

You rebuild the owner portal on owners-1

Make data survive redeploys, let containers find each other, and expose only what owners need.

1List the state and give each a home
Postgres data → volume pgdata. Photos → volume uploads for now, object storage next month. Sessions → Redis with a password.
Why: Anything not on the list can be thrown away. Anything on it needs an explicit home.

2One private network, names not localhost

All three on owners-net; DATABASE_URL=...@db:5432. The API listens on 0.0.0.0:3000 inside.

bash

docker network create owners-net
docker run -d --name db --network owners-net \
  -v pgdata:/var/lib/postgresql/data postgres:17
docker run -d --name owner-api --network owners-net \
  -e DATABASE_URL=postgres://app:…@db:5432/owners \
  -v uploads:/app/uploads -p 80:3000 owner-api:1.4.1

Why: Lina: the default bridge network doesn't do names; that's why people hard-code IPs.

3Publish only the API
Postgres and Redis get no -p at all.
Why: Publishing a database on a cloud VM means publishing it to the internet.
4Prove it survives
Upload a photo, docker rm -f owner-api, start it, fetch the photo. Then pg_dump to object storage and restore it once.
Why: Lina: a volume protects you from docker rm, not from a dead disk.

In real life

3 min

Two platforms, two answers to "where do files live?"

Heroku: the disk is wiped on restart

Publicly documents an ephemeral filesystem
Uploads go to object storage, data to a database add-on
Any instance can serve any request

Forces stateless apps from day one.

Fly.io: volumes attached to a machine

Publicly offers persistent volumes tied to one machine in one region
Good for a database or a single instance
Not shared with other machines

Local disk when needed, with the one-machine limit.

What goes wrong

3 min

Publishing database and cache ports "for convenience"

On a cloud VM that's the internet, and it bypasses ufw.

Instead

No -p for internal services; for local access, -p 127.0.0.1:6379:6379 plus a password.

App listening on 127.0.0.1 inside the container

Forwarded traffic arrives on the network interface, so it's refused even with -p.

Instead

Listen on 0.0.0.0 inside; control exposure with -p.

Treating a volume as a backup

A dead disk, deleted VM or docker volume prune takes it.

Instead

Back up off the host and test the restore.

Trade-offs

3 min

Where state lives	Fits	You pay
Named volume on the host	Dev, single-server apps, small tools	Tied to one host; backups on you
Managed services (DB, object storage)	Production with several instances	Cost, latency, vendor features
Bind mounts	Live code in development	Permission and host-path coupling

Thursday, 11:00. The portal's database.

Karim · CTO
Postgres in a container with a volume works. Managed Postgres is €180 a month more for 900 hotel owners.
Omar · On-call SRE
And who restores it at 3 a.m. when the disk dies? Who upgrades it? That's me.
Karim · CTO
Nightly backups. Worst case we lose a day of owner messages.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Where each kind of data lives: database in a volume or managed, photos in object storage, nothing in the container.
What is reachable from outside: only owner-api. PostgreSQL and Redis are never published.
Who owns backups, how often, and who tests the restore.

Tell Claude Code

Fix storage and networking for the owner portal on owners-1: owner-api, PostgreSQL 17 and Redis 7 via docker run; uploads went to /app/uploads in the container, and 212 photos were lost. Put all three on one user-defined network so owner-api uses names (db, cache), not localhost. Named volume for PostgreSQL. Don't publish PostgreSQL or Redis; Redis gets a password from an env variable. Uploads go to our object storage bucket: next month we run 3 servers. Plan first, then prove: a photo survives a redeploy, and an outside scan shows only 22, 80, 443.

Check before you accept

No -p 5432 or -p 6379 anywhere.
Connection URLs use db and cache, not localhost or an IP.
An outside port scan, not just the firewall rules.

Red flags

It fixes ECONNREFUSED by publishing the database port.
It uses a named volume for uploads several servers must share.
It hard-codes a container's IP address.

Check yourself

2 min

Q1/3Debugging

The API can't reach Postgres. What's the fix?

Both containers run on the network booking-net. The API logs: Error: connect ECONNREFUSED 127.0.0.1:5432. Its env has DATABASE_URL=postgres://app:app@localhost:5432/bookings.

How confident are you?

Next episode: a new engineer's first day. The README has 23 setup steps, and you're the one who turns them into one command.

Finished?

Mark it complete to update your skill map.

Next: Docker Compose: a whole local stack in one file

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Names, not localhost

You lead this worked example; Lina steps in at two points. The goal is the owner portal on one VM, done right.

Worked example

You rebuild the owner portal on owners-1

owner-api, PostgreSQL and Redis run on one VM with plain docker run. Make data survive redeploys, let containers find each other, and expose only what owners need.

1List the state
Write down everything that must outlive a container: Postgres's data folder (/var/lib/postgresql/data), uploaded photos (/app/uploads), owner sessions (in Redis).
Why: Anything not on this list can live in the writable layer and be thrown away. Anything on it needs an explicit home.
2Give each piece a home
Named volume pgdata for Postgres, named volume uploads for now (object storage next month), Redis with a password on the private network.
Why: A named volume survives container replacement and is managed by Docker, so it isn't tied to a host path someone might delete.
3Create a private network
docker network create owners-net and start all three containers with --network owners-net.
Why: On a user-defined network Docker answers name lookups, so the API can use db:5432 and cache:6379. Lina: "the default bridge network doesn't do names; that's why people end up hard-coding IPs."
4Use names, not localhost
Set DATABASE_URL=postgres://app:…@db:5432/owners and REDIS_URL=redis://:…@cache:6379. Make the API listen on 0.0.0.0:3000 inside its container.
bash
```
docker run -d --name owner-api --network owners-net \
  -e DATABASE_URL=postgres://app:…@db:5432/owners \
  -v uploads:/app/uploads -p 80:3000 owner-api:1.4.1
```
Why: localhost inside a container is that container. And an app listening on 127.0.0.1 inside a container can't receive forwarded traffic, even with -p.
5Publish only the front door
Only owner-api gets -p 80:3000. Postgres and Redis get no -p at all.
Why: Containers on the same network talk without published ports. Publishing a database binds it on every interface of the VM, which on a cloud VM means the internet.
6Prove it survives
Upload a test photo, run docker rm -f owner-api, start it again, fetch the photo. Then pg_dump to object storage and restore it once on a test VM.
Why: Lina: "a volume protects you from docker rm, not from a dead disk. A backup you haven't restored is a hope."

	Named volume	Bind mount	tmpfs
Managed by	Docker (/var/lib/docker/volumes)	You (any host path)	Kernel memory
Survives container removal	Yes	Yes (it's your folder)	No
Portable across hosts	No (local to the host) unless a volume driver is used	No (depends on host paths)	N/A
Performance on macOS/Windows	Good (lives inside the Docker VM)	Slower for many small files	Fastest
Typical use	Database files, local persistent data	Source code in dev, config files	Scratch space, secrets at runtime

Symptom	Most likely cause	Check
`could not translate host name "db"`	Containers on different networks, or on the default bridge (no DNS)	docker network inspect
`connection refused` to `localhost:5432` from the API	localhost is the API container itself	Use the name: db:5432
`connection refused` from the host, container running	No -p, or app listens on 127.0.0.1 inside the container	docker port; listen on 0.0.0.0
`port is already allocated`	Another process or container holds that host port	Map a different host port (-p 3001:3000)
Permission denied writing to a bind mount	Container UID differs from the host folder owner	Match UIDs or use a named volume

The error message usually tells you which part is wrong: name, listener, publish or permissions.

Interactive simulation

Visual diagram

Your browser

Host: port 8080 published

api (listens :3000)

Embedded DNS 127.0.0.11

db (Postgres :5432)

Volume pgdata

cache (Redis :6379)

Click a component

One host, one private network, one published port. Click components or start the walkthrough.

Analogy

A hotelDocker

A hotelA room, fully cleaned between guestsDockerA container's temporary files
A hotelThe luggage storage roomDockerA named volume
A hotelBringing your own suitcaseDockerA bind mount from your laptop
A hotelRooms calling each other by nameDockerContainers on a shared network
A hotelThe front desk's public numberDockerA published port

Where the analogy breaks: a hotel's storage room serves the whole building. A Docker volume belongs to one machine, so it can't share data between servers and isn't a backup.

Check your understanding

Q1/3Debugging

docker run -d -p 8080:3000 booking-api is running, logs say listening on 127.0.0.1:3000, but curl localhost:8080 from the host is refused. Why?

How confident are you?

Practice ladder

Step 1 · Trace

Trace three containers on owners-1

Scenario Warm-up

On owners-1:

docker network create owners-net
docker run -d --name db --network owners-net -v pgdata:/var/lib/postgresql/data postgres:17
docker run -d --name cache redis:7
docker run -d --name owner-api --network owners-net -p 80:3000 \
  -e DATABASE_URL=postgres://app:…@db:5432/owners \
  -e REDIS_URL=redis://cache:6379 owner-api:1.4.1

The API writes uploads to /app/uploads. Predict: 1. Can the API reach the database? Redis? 2. Can someone on the internet reach Postgres? 3. After docker rm -f owner-api and starting it again, what survives: database rows, uploads?

Step 2 · Complete

Order the checks: the API can't reach Redis

Debugging Practice

owner-api logs Error: getaddrinfo ENOTFOUND cache. Lina's checklist is scrambled. Put these checks in order, from cheapest and most likely to least, and say what each would tell you:

A. Check Redis's password in the API's REDIS_URL
B. docker ps: is the cache container running?
C. docker network inspect owners-net: are both containers listed?
D. Rebuild the owner-api image
E. docker exec owner-api getent hosts cache: does the name resolve?

Step 3 · Solve

Make the reviews VM safe

Design Practice

The reviews service runs on one VM:

docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=reviews123 postgres:17
docker run -d -p 6379:6379 redis:7
docker run -d -p 80:3000 \
  -e DATABASE_URL=postgres://postgres:reviews123@172.17.0.2:5432/postgres reviews-api:2.0.3

reviews-api stores guest review photos in /app/media. Rewrite the setup so that data survives redeploys, containers find each other by name, only the API is public, and there's a backup off the VM. List each change and why.

Step 4 · Decide

Session store for three API servers

Decision Stretch

Next month owner-api runs on app-1, app-2 and app-3 behind the load balancer. Sessions are now in Redis on owners-1. Two proposals:

A (Karim): run a Redis container on each app server, with a named volume. "No network hop, and it's free."
B (Omar): one managed Redis that all three servers use. About €40 a month.

The load balancer spreads requests evenly, with no sticky sessions.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain how a request from your browser reaches the API container and how the API then reaches Postgres, and where the database's data actually lives.

0 words