Episode 62 · The six-minute one-line fix

Dockerfiles and layers: fast builds, small and safe images

Wednesday, 11:05: the search team fixes a one-character typo and waits 6 minutes 10 seconds for CI. Their image weighs 1.4 GB, and the security scanner just flagged 300 problems in it.

A few habits in your Dockerfile turn slow, huge, risky images into fast, small, safe ones.

30 min intermediateLayer cache keys and instruction orderingBuild context and .dockerignoreSlim, distroless and non-root imagesDockerfile & layersImages vs containers

Problem
The search API image is 1.4 GB, every CI build takes 6 minutes even for a one-line fix, a new server needs about a minute just to pull the image, and the security scanner reports 300 vulnerabilities.
Why the simple system fails
A naive Dockerfile copies everything first, installs build tools and dev dependencies into the final image, runs as root, and sends the whole repo (including .git and .env) to the builder.
Engineering concept
Treat the Dockerfile as a cache-aware build graph: order steps from least to most frequently changing, build in one stage and run from a minimal one, and ship only what the process needs, as a non-root user.
Technology
BuildKit, layer cache keys, cache mounts, multi-stage builds, slim and distroless bases, USER, exec-form CMD, .dockerignore, image scanners.
Trade-offs
Faster, smaller, safer images in exchange for a more complex Dockerfile, harder interactive debugging (no shell), and occasional libc or tooling surprises with minimal bases.

The problem

2 min

Wednesday, 11:05. The search team's first image is in CI.

Maya · Product manager
We fixed a typo. CI says 6 minutes 10 seconds.
Omar · On-call SRE
And the image is 1.4 GB. A new server needs a minute just to download it.
Lina · Senior engineer, your mentor
Rebuilding with no change takes 10 seconds. So the machine isn't slow. Guess what is.

The big idea

4 min

Why a one-line fix rebuilds everything

The big idea

The first changed step rebuilds everything after it. Put what changes most, last.

Like glued sheets: to change sheet 3, you peel off everything on top. Speed is one problem; size is another, with its own fix: ship only what running needs.

How it works

5 min

How Docker decides what to reuse, and what to ship

1
Layers and the build cache
Each step adds a layer. The build cache reuses a layer if its step and every step above it are unchanged: a from the parent and the instruction.
2
First miss breaks the chain
For COPY, the key includes the files' checksum. Our typo fix changes COPY . ., so npm install (4.5 min) and the build re-run every time.
3
Reorder
Copy only package.json and the lockfile, install, then copy the code. Code changes now stop after the install: about 50 s.
4
Trim the build context
The build context is the folder sent to Docker: 900 MB here. A .dockerignore keeps out .git, node_modules, .env*.
5
Multi-stage: build fat, run thin
A builds in a full stage, then copies only dist and production node_modules into a slim one. TypeScript and test tools stay behind: 1.4 GB → about 220 MB.
6
Layers only add
RUN rm in a later step hides files; it never shrinks the image. Keep files out, or leave them in the build stage.

Watch a decision being made

4 min

Worked example

Lina and you rewrite the search-api Dockerfile

1.4 GB, 6 min 10 s per change, ~300 scanner findings. Fast builds, a small image, a safe process, no app changes.

1Measure the image
docker history: full Debian 650 MB, dev dependencies 480 MB, .git and test data 170 MB, our code 20 MB.
Why: Most of the weight is build-only. You can't shrink what you haven't measured.
2Trim, then order for the cache
.dockerignore for .git, node_modules, .env*; lockfile and npm ci before COPY . ..
Why: A code change now breaks the chain after the install.

3Split build and run

Stage 1 builds and prunes dev dependencies; stage 2 copies only the output.

dockerfile

FROM node:22 AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev
FROM node:22-slim
WORKDIR /app
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER node
CMD ["node", "dist/server.js"]

Why: The final image is only its own stage.

4Run safely, measure again
USER node, exec-form CMD. Result: ~45 s builds, 220 MB, ~40 findings, new servers pull in ~9 s instead of ~56 s.
Why: Non-root limits damage; exec form lets node hear SIGTERM. Measuring proves the fix.

In real life

3 min

Two well-known base image choices

Docker Official Images (node:22-slim)

The most common start: a small Debian with a shell and package manager
Easy to debug: open a shell in a running container
More packages, more scanner findings

Fits teams that still debug inside containers.

Google's distroless images

A public Google project: only the runtime, no shell, no package manager
Far fewer findings, nothing for an attacker to use
Debugging needs logs, metrics or a debug image

Fits when attack surface matters and debugging works without a shell.

What goes wrong

3 min

A build-specific `ARG` (commit SHA) near the top

It changes the cache key of every later step: every build reinstalls.

Instead

Declare it as late as possible, or use labels at the end.

Cleaning up in a separate `RUN`

The files stay in the layer that created them.

Instead

Create and clean in the same RUN, or use a build stage.

Distroless without a debugging plan

At 3 a.m. there's no shell, and the team reverts to a fat image.

Instead

A debug build target, plus good logs and metrics first.

Trade-offs

3 min

Approach	Fits	You pay
Single-stage, full base	Prototypes, a first containerisation	Big images; many packages to patch
Multi-stage, slim runtime	A sensible production default	Still ships a shell and package manager
Multi-stage, distroless	Internet-facing or regulated services	No shell for debugging

Wednesday, 15:00. Sprint planning.

Karim · CTO
Two days to rewrite a Dockerfile that works? The feature is due Friday. Six minutes is a coffee break.
Lina · Senior engineer, your mentor
About 40 builds a day: over three hours of waiting on CI, daily.
Karim · CTO
People do other things while CI runs.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The targets: ~220 MB instead of 1.4 GB, code-change builds under a minute.
Which runtime base to accept, knowing smaller means fewer debugging tools.
How the container stops: exec-form CMD, so the app drains on SIGTERM.

Tell Claude Code

Restructure the search-api Dockerfile. Today: FROM node:22, copies everything before npm install, ships TypeScript and dev deps, runs as root, no .dockerignore, 1.4 GB. I want a .dockerignore excluding .git, node_modules and .env files; a multi-stage build installing from the lockfile before copying source; a slim Node 22 runtime with only compiled output and production deps; a non-root user; exec-form CMD. Don't change app code. Show image size, one-line-change build time, cached steps, and docker stop timing during a search.

Check before you accept

In the code-change rebuild log, the install step says CACHED.
The last line is exec form, and docker stop takes about a second, not exactly 10.
whoami inside the container is not root.

Red flags

It deletes files in a later RUN and claims the image got smaller.
It passes secrets through ARG or ENV.
It switches back to root to fix a permission error.

Check yourself

2 min

Q1/3Debugging

CI passes --build-arg GIT_SHA=$(git rev-parse HEAD). The install step never hits the cache, even when the lockfile hasn't changed. Why?

FROM node:22-slim
ARG GIT_SHA
ENV APP_VERSION=$GIT_SHA
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY . .
CMD ["node", "dist/server.js"]

How confident are you?

Next episode: hotel owners' uploaded photos vanish after a redeploy, and someone finds StayHub's Redis open to the whole internet.

Finished?

Mark it complete to update your skill map.

Next: Volumes and networks: where container data lives and how containers talk

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

From 1.4 GB to 220 MB

Lina asks you to try the first step yourself, then takes over. Here's the full rewrite of the search API's Dockerfile, step by step.

Worked example

Lina and you rewrite the search-api Dockerfile

search-api's image is 1.4 GB, every code change takes 6 min 10 s to build, and the scanner reports ~300 findings. Make builds fast, the image small and the process safe, without changing the app.

1Measure the image
Run docker history search-api:1.0.0 and note the biggest layers: the full Debian base (650 MB), dev dependencies (480 MB), .git and test data (170 MB).
Why: You can't shrink what you haven't measured. The list also shows that most of the weight is build-only.
2Shrink the build context
Add a .dockerignore so .git, node_modules, .env* and test fixtures never reach the builder.
text
```
.git
node_modules
.env*
test/fixtures
*.log
```
Why: Less to send to the builder, fewer files that can change a COPY checksum, and no secrets copied by accident.
3Order for the cache
Copy package.json and the lockfile, run npm ci, and only then COPY . . and build.
Why: Now a code change breaks the chain after the 4.5-minute install. Builds for code changes drop to about 50 s.
4Split build and run
Stage 1 (FROM node:22 AS build) installs everything, builds and prunes dev dependencies. Stage 2 (FROM node:22-slim) copies only node_modules and dist.
dockerfile
```
FROM node:22-slim
WORKDIR /app
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
```
Why: The final image is only its own stage. TypeScript, test tools and the Debian toolchain stay behind: 1.4 GB becomes about 220 MB.
5Run safely
End with USER node and the exec form CMD ["node", "dist/server.js"].
Why: A non-root user limits damage if the app is compromised. Exec form makes node the first process, so it receives SIGTERM and can drain requests on deploy.
6Measure again
Rebuild after a one-line change: about 45 s. docker images: 220 MB. Scanner: about 40 findings. A new server pulls in about 9 s.
Why: Measuring again proves the fix and shows what's left: the remaining findings are mostly in the slim base, which a weekly rebuild keeps patched.

Base image family	Typical size	Shell / package manager	libc	Good for
Full distro (debian, ubuntu)	300 MB – 1 GB+ with runtime	Yes / yes	glibc	Build stages, legacy apps needing many OS tools
Slim variants	~70–250 MB with runtime	Yes / yes	glibc	A safe default runtime; easy debugging
Alpine	~5 MB base, ~50–150 MB with runtime	Yes / apk	musl	Small images when native modules are known to work with musl
Distroless	~20 MB base + runtime	No / no	glibc	Hardened production runtimes
scratch	0 MB	No / no	None	Static binaries (Go, Rust) only

Illustrative sizes; they change with versions. Measure your own image with docker images and docker history.

Interactive simulation

Visual diagram

Repo (build context)

Stage deps: lockfile + install

Stage build: compile + test

BuildKit cache

Stage runtime (distroless, non-root)

Registry

Click a component

A multi-stage build: what gets cached, what gets shipped. Click components or start the walkthrough.

Analogy

A cake factoryA Dockerfile

A cake factoryThe kitchen with ovens and mixersA DockerfileThe build stage with all the tools
A cake factoryThe delivery box with just the cakeA DockerfileThe small runtime image
A cake factoryReusing yesterday's spongesA DockerfileReusing cached layers
A cake factoryChanging the flour redoes everythingA DockerfileAn early change rebuilds all later steps
A cake factoryNot loading the whole warehouse in the vanA Dockerfile.dockerignore keeps junk out

Where the analogy breaks: you can scrape icing off a cake. In an image, removing something in a later layer doesn't make the earlier layers lighter.

Check your understanding

Q1/3Design

Order these Dockerfile steps of a build stage from first to last for the best cache reuse:

1.COPY requirements.txt .
2.FROM python:3.12-slim
3.COPY app/ ./app
4.RUN pip install -r requirements.txt
5.RUN apt-get update && apt-get install -y libpq-dev && rm -rf /var/lib/apt/lists/*

How confident are you?

Practice ladder

Step 1 · Trace

Which steps re-run?

Scenario Warm-up

Here's the fixed search-api build stage:

FROM node:22 AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

For each change below, list which steps re-run and estimate the build time. Install takes ~4.5 min, build ~40 s, everything else a few seconds.

You edit src/search.ts.
You add a new package with npm install fastify-cors.
You edit README.md (not in .dockerignore).

Step 2 · Complete

Finish the final stage

Code Practice

Lina wrote the build stage of the reviews-api (TypeScript, built to dist/, started with node dist/server.js, port 3000). Write the final stage so that:

it uses a slim Node 22 base,
it contains only production dependencies and dist/,
it runs as a non-root user,
node receives SIGTERM on deploy.

FROM node:22 AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev

# your final stage here

Step 3 · Solve

The cache that never hits in CI

Debugging Practice

The pricing-worker image builds in 20 s on laptops after a code change. In CI it takes 5 min 40 s every time, even for code-only changes. The Dockerfile starts like this:

FROM python:3.12-slim
ARG GIT_SHA
ENV APP_VERSION=$GIT_SHA
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

CI runs docker build --build-arg GIT_SHA=$(git rev-parse HEAD) . on a fresh VM for every job.

Find both causes and fix them.

Step 4 · Decide

Distroless for the payments service?

Decision Stretch

The payments service (Node 22) handles card tokens. The facts:

A security review before StayHub's fundraising asks for zero critical findings on internet-facing services. The slim image has 3 critical, fixable ones in OS packages it doesn't use.
On-call opened a shell in a payments container twice last quarter to inspect files.
The service has good structured logs and metrics.
A distroless switch takes about a day, including a debug build target.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

A teammate asks why their image is 1.4 GB and why every build reinstalls dependencies. Explain what's happening and how you'd restructure the Dockerfile.

0 words