Skip to content
Episode 65 · The server that died at 3:12

Kubernetes explained

Tuesday, 3:12 a.m.: node-5 dies, taking all three search containers with it. Search stays down for 47 minutes, until Omar wakes up and restarts them by hand on other servers.

You tell Kubernetes what should be running. It keeps checking, and fixes anything that drifts. Here's how, and when it's worth it.

30 min advancedDesired state & reconciliationPods, deployments, servicesKubernetes services & ingressConfigMaps & SecretsHealth checks (liveness/readiness)Autoscaling
  1. Problem

    You have 40 containers across 8 machines. One crashes at 3 a.m., traffic triples on Friday, and every release must roll out without downtime.

  2. Why the simple system fails

    Scripts and SSH can start containers, but they can't continuously notice a dead one, move work off a failing machine, or replace versions gradually and safely.

  3. Engineering concept

    Declare the desired state ("3 replicas of booking-api:3.2.0, each with 0.5 CPU"); controllers compare it with reality and act until they match, forever.

  4. Technology

    Kubernetes: cluster, nodes, pods, deployments, services, ingress, ConfigMaps/Secrets, probes, the Horizontal Pod Autoscaler, rolling updates.

  5. Trade-offs

    Enormous flexibility and portability, paid for with significant complexity and operational expertise. Small teams often get 90% of the value from a PaaS.

01

The problem

2 min
Tuesday, 10:00. The postmortem for last night.
  1. Omar · On-call SRE
    03:12 node-5 died. All 3 search containers were on it. I woke up at 03:51, search was back at 03:59. 47 minutes.
  2. Maya · Product manager
    About 1,900 searches failed. Mostly guests in Asia, where it was morning.
  3. Karim · CTO
    We run 40 containers on 8 servers. This will happen again. What stops it?
02

The big idea

4 min
Who restarts the containers at 3 a.m.?
The big idea

Declare what should exist. A loop keeps comparing it with reality and fixing the difference.

Our scripts start containers once and exit; nothing watches after that. works like a thermostat: you set the target and it keeps correcting, all night. Same dead server, about 2 minutes of downtime, and nobody wakes up.

03

How it works

5 min
How the control loop runs a cluster
  1. 1
    You declare

    kubectl apply sends "search-api: 3 replicas of 2.4.1, 0.5 CPU each" to the cluster's API, which stores this desired state. A is the object that keeps a set number of identical pods running (a pod is the smallest thing Kubernetes runs: one or a few containers sharing an address).

  2. 2
    Controllers observe

    A controller is a loop that counts what really exists. Each node reports which pods it runs and whether it's alive.

  3. 3
    They compare

    Desired 3, actual 3: nothing to do. node-5 stops reporting for about 40 s: it's marked unhealthy and its pods are lost. Actual 0.

  4. 4
    They act

    The controller creates 3 new pods. The scheduler places them on nodes with enough free CPU and memory, judging by each pod's requests (what it says it needs).

  5. 5
    Traffic follows

    A is one stable name in front of pods whose IPs keep changing. It only sends traffic to pods that pass their readiness check.

  6. 6
    The same loop scales

    An changes the replica count: ceil(pods × usage ÷ target). booking-api at 4 pods, 90% CPU, target 60%: it asks for 6.

04

Watch a decision being made

4 min
Worked example

Lina declares booking-api on Kubernetes

booking-api runs as 3 containers pinned to node-1..3 by a script. Move it so it survives a node failure, keeps one stable address, and scales on Friday evenings.

  1. 1Declare the copies, with their size

    A Deployment: 3 replicas of booking-api:3.2.0, with requests: { cpu: 500m, memory: 512Mi } measured on today's servers, and a memory limit of 768Mi.

    Why: The scheduler places pods by their requests. Without them it places blind, and the autoscaler can't compute CPU usage.

  2. 2Give it one stable name

    A Service booking-api that selects app: booking-api and forwards port 80 to 3000.

    yaml
    kind: Service
metadata: { name: booking-api }
spec:
  selector: { app: booking-api }
  ports: [{ port: 80, targetPort: 3000 }]

    Why: Pods get a new IP every time they're recreated. Callers use one name and never care which pods are behind it.

  3. 3Say when a pod is ready, and let it scale

    A readiness probe on /ready (200 once the database pool is open), and an autoscaler: min 3, max 20, target 60% CPU.

    Why: A starting pod gets no traffic it can't serve. Friday evenings are about 3× a Tuesday morning.

  4. 4Break it on purpose

    In staging, drain one node and watch: pods are recreated elsewhere, the Service keeps answering.

    Why: The promise is only real once you've watched it hold.

The whole thing in one line: Lina declared what should exist (sized copies, a stable name, a readiness rule, a scaling target) and let the controllers keep reality matching it. A dead node or a busy Friday no longer needs a human.
05

In real life

3 min
Build your own orchestrator, or adopt the standard?

Netflix: its own container platform

  • Has publicly described Titus, its own container platform built on top of AWS
  • Tightly fitted to its huge, AWS-specific infrastructure
  • Needs a dedicated team to build and run it

Worth it at enormous scale with very specific needs.

Spotify: moved to Kubernetes

  • Has publicly described moving from its own orchestrator (Helios) to Kubernetes
  • Trades a custom tool for a standard with a large ecosystem
  • Hiring and tooling get easier; migration takes years

Worth it when a standard tool covers most of your needs.

06

What goes wrong

3 min
No resource requests

The scheduler places pods blind, nodes get overfilled, and the autoscaler has nothing to measure against.

Instead

CPU and memory requests from measured usage, plus a memory limit.

Liveness probe checks the database

A 30-second database blip fails every pod at once; Kubernetes restarts them all, turning a blip into an outage.

Instead

Liveness checks only the process itself.

Adopting Kubernetes for one app

Cluster upgrades, networking and security patches become a part-time job for a team that should ship product.

Instead

Use a PaaS until services, teams or special workloads justify a platform.

07

Trade-offs

3 min
OptionFits whenYou pay
PaaS (Render, Cloud Run…)A few services, standard web workloadsHigher unit cost at scale, platform limits
Managed KubernetesMany services and teams, special workloadsLearning curve, owners, upgrades
Self-managed KubernetesOn-premises or regulated, expert teamYou run the control plane and its backups
Tuesday, 15:00. Karim wants a decision this week.
  1. Karim · CTO
    A PaaS restarts dead containers too. No cluster to upgrade, no YAML.
  2. Lina · Senior engineer, your mentor
    For 3 services I'd agree. We have 14, plus nightly batch jobs and a memory-hungry worker.
  3. Karim · CTO
    And who runs the cluster? I'm not hiring a platform team.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • Whether StayHub needs Kubernetes at all, and managed or self-run. Here: yes, managed, with a 2-person platform team.
  • The resource requests and limits for every workload (booking-api: 500m CPU / 512Mi). Without them the scheduler is guessing.
  • Autoscaling rules: which metric, which target (60% CPU), minimum and maximum replicas.
Tell Claude Code
Write Kubernetes manifests for search-api so one dead node can never take it down. A Deployment with 3 replicas spread across nodes, requests of 500m CPU and 512Mi with a memory limit, readiness on /ready, liveness on /healthz that doesn't check the database, an HPA at 60% CPU (min 3, max 10), and a Service. Image search-api:2.4.1, not latest. Explain each field in one line, apply to staging only, drain one node and show how long until 3 pods are ready.
Check before you accept
  • Every container has CPU and memory requests.
  • The 3 replicas land on different nodes (kubectl get pods -o wide).
  • The drill result: about 1–2 minutes until 3 pods are ready again, not 47, and nothing applied to production.
Red flags
  • It leaves out resource requests to "let Kubernetes decide".
  • It "stops" something by deleting pods instead of changing the desired state.
  • It puts the database in the cluster as a plain Deployment.
09

Check yourself

2 min
Q1/3Scenario
A Deployment declares 3 replicas. You run kubectl delete pod booking-api-7f9c-x2k to 'turn it off'. What happens?
How confident are you?

Next episode: booking-api's pods are all Running and Ready, yet every call to them times out.

Finished?

Mark it complete to update your skill map.

Next: Pods, Deployments and Services in depth
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

One Deployment, one Service, one autoscaler

Lina moves booking-api to the cluster first, thinking out loud. Every object below is declarative: you describe the end state and kubectl apply it.

Worked example

Lina declares booking-api on Kubernetes

booking-api runs as 3 containers pinned to node-1..3 by a script. Move it to the cluster so it survives a node failure, keeps one stable address, and scales on Friday evenings.

  1. 1Declare the copies

    Write a : 3 replicas of booking-api:3.2.0, with the label app: booking-api on every pod.

    Why: The Deployment is the promise the loop keeps. The exact image tag means every replica runs the same thing, as in the Docker module.

  2. 2Say what each pod needs

    Add requests: { cpu: 500m, memory: 512Mi } (measured on today's servers) and limits: { memory: 768Mi }.

    Why: The scheduler places pods by their requests. Without them it places blind, and the autoscaler can't compute CPU usage.

  3. 3Give it one stable name

    Add a booking-api that selects app: booking-api and forwards port 80 to 3000.

    yaml
    kind: Service
metadata: { name: booking-api }
spec:
  selector: { app: booking-api }
  ports: [{ port: 80, targetPort: 3000 }]

    Why: Pods get a new IP every time they're recreated. Callers use booking-api and never care which pods are behind it.

  4. 4Tell it when a pod is ready

    Add a readiness probe on /ready, which returns 200 once the database connection pool is open.

    Why: The Service only sends traffic to ready pods, so a starting pod doesn't get requests it can't serve. Episode 4 goes deep on this.

  5. 5Let it scale

    Add a : min 3, max 20, target 60% average CPU.

    Why: Friday evenings are about 3× a Tuesday morning. The autoscaler raises replicas when usage climbs, using ceil(current × usage / target).

  6. 6Break it on purpose

    In staging, drain one node (kubectl drain node-2) and watch: pods are recreated elsewhere, the Service keeps answering.

    Why: The promise is only real once you've watched it hold. It also shows whether 3 replicas leave enough room when one node is gone.

The whole thing in one line: Lina declared what should exist (3 sized copies, a stable name, a readiness rule, a scaling target) and let Kubernetes' controllers keep reality matching it, so a dead node or a busy Friday no longer needs a human.
bash
kubectl apply -f booking-api.yaml                 # declare desired state
kubectl get pods -l app=booking-api -o wide       # see actual state, and on which nodes
kubectl set image deploy/booking-api api=registry.stayhub.dev/booking-api:3.2.1
kubectl rollout status deploy/booking-api         # watch the rolling update
kubectl rollout undo deploy/booking-api           # back to the previous version
kubectl get hpa booking-api                       # current vs target CPU, replicas
Day-to-day commands.
ProbeQuestion it answersOn failureShould check
ReadinessCan this pod serve traffic right now?Removed from Service endpoints, not restartedApp initialised; can reach its critical dependencies
LivenessIs this process hopelessly stuck?Container is restartedOnly the process itself. Never external dependencies
StartupHas a slow-starting app finished booting?Holds off liveness until it passesSame as liveness, with a longer budget
11

Interactive simulation

12

Visual diagram

Click a component

Traffic and control in a small cluster running the booking API.

13

Analogy

  • Running a hotel's front deskThe staffing plan: 6 receptionistsKubernetesDesired state
  • Running a hotel's front deskThe operations manager checking the floorKubernetesThe control plane
  • Running a hotel's front deskBuildings, and staff placed in themKubernetesNodes and pods
  • Running a hotel's front deskOne front-desk phone numberKubernetesA Service
  • Running a hotel's front deskRetraining staff two at a timeKubernetesA rolling update
  • Running a hotel's front deskCalling in extra staff when queues growKubernetesAutoscaling

Where the analogy breaks: a manager uses judgement. Kubernetes follows your rules exactly, even bad ones. If you tell it to restart pods whenever the database is slow, it will, all of them.

14

Check your understanding

Q1/4Debugging
The database had a 40-second failover. During it, all 12 booking-api pods restarted repeatedly and the outage lasted 6 minutes. Most likely cause?
livenessProbe:
  httpGet: { path: /health, port: 3000 }   # /health runs SELECT 1 on Postgres
  periodSeconds: 5
  failureThreshold: 2
How confident are you?
15

Practice ladder

Step 1 · Trace

Trace a node failure

Scenario Warm-up

The cluster has 4 nodes. search-api is a Deployment with 3 replicas; its pods run on node-1, node-3 and node-3 (two on the same node). Each pod requests 0.5 CPU (500m). Free CPU: node-1: 0.3, node-2: 1.2, node-4: 0.2 (node-3's free CPU no longer matters).

At 03:12 node-3 dies. Trace what happens over the next few minutes: what the controller sees, where new pods can go, and what callers of the search-api Service experience.

Step 2 · Complete

Finish Lina's Deployment

Code Practice

Lina's Deployment for the reviews-api has three gaps. Measured usage: about 300m CPU and 350 MiB of memory per pod, peaks at 500 MiB.

apiVersion: apps/v1
kind: Deployment
metadata: { name: reviews-api }
spec:
  replicas: 3
  selector: { matchLabels: { app: reviews-api } }
  template:
    metadata: { labels: { app: ___1___ } }
    spec:
      containers:
        - name: api
          image: ___2___
          resources:
            ___3___

Fill in the gaps and say in one line why each matters.

Step 3 · Solve

Size the Friday autoscaler

Estimation Practice

booking-api has an autoscaler: min 3, max 20, target 60% CPU. Each pod requests 500m CPU. Omar's numbers:

  • Tuesday 10:00: 3 pods at 40% average.
  • Friday 19:00: traffic is 3× Tuesday's.
  • Each node has 4 CPUs free for booking-api.
  1. Roughly how many pods does the autoscaler want on Friday evening?
  2. How many nodes' worth of room is that?
  3. What would happen if someone set the request to 2 CPUs "to be safe"?
Step 4 · Decide

Put the nightly reports in the cluster?

Decision Stretch

The nightly revenue report (from the Docker module) runs as a container on its own VM, reports-1, once a night for 25 minutes. It needs 2 CPUs and 3 GB of memory while running. The cluster has spare capacity at night (booking traffic is lowest between 02:00 and 05:00).

Omar: "Make it a Kubernetes CronJob in the cluster and delete reports-1." Karim: "Why touch something that works? It's €60 a month."

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain what Kubernetes does when a pod crashes and when traffic triples, and when you would advise a team not to adopt it.

0 words