Episode 66 · Running, Ready, and unreachable

Pods, Deployments and Services in depth

Thursday, 14:05: all six booking-api pods are Running and Ready, yet every checkout call to them times out. In 11 minutes, about 420 guests can't pay.

"It deployed but nothing works" usually means the pieces can't find each other. Here's how they connect, through simple name tags.

30 min advancedLabels, selectors and Deployment → ReplicaSet → Pod ownershipService types: ClusterIP, NodePort, LoadBalancerNamespaces: scoping names, quotas and permissionsPods, deployments, servicesKubernetes services & ingressDesired state & reconciliation

Problem
Your booking API runs as pods whose IPs change on every restart, while three other services, a public website and a partner API all need to reach it reliably.
Why the simple system fails
Hard-coding pod IPs breaks on the next restart; one cloud load balancer per service gets expensive and messy; one shared namespace lets a test job starve production of CPU.
Engineering concept
Everything is wired by labels: Deployments own ReplicaSets that own Pods, Services select ready Pods by label, Ingress routes HTTP to Services, and namespaces scope names, quotas and permissions.
Technology
Pod spec, ReplicaSet, Deployment, label selectors, ClusterIP / NodePort / LoadBalancer Services, EndpointSlices, Ingress + an ingress controller, Namespaces + ResourceQuota.
Trade-offs
Loose coupling through labels is flexible but fails silently on a typo; LoadBalancer Services are simple but cost one cloud LB each; namespaces organise but don't isolate without RBAC and network policy.

The problem

2 min

Thursday, 14:05. booking-api's chart was refactored this morning.

Maya · Product manager
Checkout is failing. About 40 guests a minute can't pay.
Omar · On-call SRE
booking-api: 6/6 Running, 6/6 Ready. No restarts. Checkout's calls just time out.
Lina · Senior engineer, your mentor
You first this time. The pods are healthy. What else is on the path?

The big idea

4 min

Six healthy pods, zero endpoints

The big idea

Kubernetes objects find each other by label, not by name or IP. A label typo fails silently: check the endpoints first.

Thursday's fix was one line: put the label back. Endpoints returned within seconds, but only after 11 minutes and about 420 failed payments. Loose coupling by labels is what makes Kubernetes flexible, and why no object ever reported an error.

How it works

5 min

From a guest's click to a pod

1
Deployment owns versions
A creates one ReplicaSet per version; the ReplicaSet keeps the right number of pods. Old ReplicaSets stay, scaled to zero, so rollout undo just scales one back up: about 30 s, no build.
2
Pods carry labels
A label is a name tag like app: booking-api. The Deployment stamps it on every pod it creates.
3
Service selects by label
A carries a selector, a search for labels. It matches pods, not the Deployment's name.
4
Endpoints: ready matches only
Kubernetes lists every pod that matches and is ready. A call to booking-api (port 80) goes to one of them on port 3000. No match: no endpoints, calls go nowhere.
5
One front door
An Ingress routes outside HTTP traffic by host and path to internal Services. It needs a controller, behind one cloud load balancer. Internal Services stay ClusterIP.
6
Namespaces group, they don't wall
A namespace scopes names, quotas and permissions. It doesn't block traffic: any pod can call booking-api.bookings unless a says no.

Watch a decision being made

4 min

Worked example

Lina wires search-api so a typo can't cut it off

Move search-api into the search namespace so services call it by name, guests reach it at api.stayhub.com/search, and a label typo can't silently cut it off.

1Define the label once
app: search-api lives in one chart value, used by the Deployment's selector, the pod template and the Service's selector.
Why: Thursday's outage was a label drifting in one of three places. One source of truth removes that class of bug.
2Keep it internal
Service type ClusterIP, port: 80, targetPort: http (a named container port). Callers use search-api.search.
Why: "Nothing internal gets a public load balancer. Fewer front doors, fewer to secure and pay for."
3Route outside traffic through the shared Ingress
One rule: host api.stayhub.com, path /search → Service search-api, port 80. Check the Ingress shows an ADDRESS.
yaml
```
rules:
  - host: api.stayhub.com
    http:
      paths:
        - path: /search
          pathType: Prefix
          backend: { service: { name: search-api, port: { number: 80 } } }
```
Why: An Ingress is only a request; the controller routes. No ADDRESS means no controller picked it up.
4Verify from the inside out
Endpoints exist? A curl from a throwaway pod to http://search-api.search/ready? Then from outside to https://api.stayhub.com/search.
Why: Each check rules out one layer in seconds.

In real life

3 min

Who finds the instances: the caller or the platform?

Netflix: the caller looks them up (Eureka)

Publicly open-sourced Eureka, a registry that instances report to
Each caller fetches the list and picks an instance itself
Flexible, but every service needs the client library

Fits a large fleet built before platform-level discovery existed.

Kubernetes Services: the platform does it

Callers use one stable name; the platform keeps the list of ready pods
Works for any language, no library
Spreads connections, not requests, so long-lived connections stick to one pod

Fits most teams already on Kubernetes.

What goes wrong

3 min

Treating namespaces as a security boundary

Staging pods can call production Services, and cluster-wide roles cross namespaces.

Instead

Default-deny network policies, permissions per namespace, and a separate cluster for staging.

Selector and pod labels don't match

Zero endpoints; callers time out. No object reports an error.

Instead

Check kubectl get endpointslices first; define labels once.

A LoadBalancer Service for every microservice

Each one is a monthly bill, a public IP to secure and TLS to manage.

Instead

ClusterIP inside; one Ingress controller for HTTP entry.

Trade-offs

3 min

Option	Gains	Costs	Choose when
One LoadBalancer per service	Nothing new to run; failures stay per service	~€20/month each, many public IPs and certificates	A handful of services, or raw TCP/UDP
One Ingress controller	One LB, TLS and routing in one place	A controller to run; a bad rule affects everyone	Several HTTP services behind one domain

Wednesday, 11:00. Twelve HTTP services need outside access.

Karim · CTO
Give each one type: LoadBalancer. Works today, no new component.
Omar · On-call SRE
That's €240 a month, 12 public IPs and 12 TLS certificates to keep valid.
Karim · CTO
And your Ingress is one more thing to run. If it breaks, everything's unreachable.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The label convention every chart follows (app: booking-api). A label rename is a breaking change, reviewed like an API change.
How each service is exposed: ClusterIP by default, one Ingress for the 12 HTTP services (~€20/month) instead of a load balancer each (~€240/month).
What separates staging from production: network policies and permissions, since namespaces are not walls.

Tell Claude Code

Refactor the booking-api Helm chart to use our shared label helpers, but the Service selector and pod labels must keep matching exactly: app: booking-api. Don't change replicas (6), ports (3000/80) or the namespace. Render the chart first and show me the selector and pod labels side by side, before and after. Apply to staging and show 6 ready endpoints. Add a NetworkPolicy so staging pods can't reach bookings, and show me how you tested it blocks traffic.

Check before you accept

The Service selector and the pod template labels match, letter by letter, in the rendered YAML.
After applying: 6 ready endpoints for 6 ready pods.
A request from a staging pod to booking-api.bookings.svc fails.

Red flags

It renames labels for tidiness without checking every selector that uses them.
It reports "all pods Running and Ready" as proof traffic works.
It exposes an internal service with its own LoadBalancer "for easier testing".

Check yourself

2 min

Q1/3Debugging

All 6 booking-api pods are Running and Ready, but calls to the Service time out. kubectl get endpointslices shows no endpoints. What's the most likely cause?

# Service
selector: { app: booking-api }
# Pod template
labels: { app: bookings-api, tier: api }

How confident are you?

Next episode: the payment key must be rotated by 18:00, and it turns out it sits in git, base64-encoded, readable by 140 people.

Finished?

Mark it complete to update your skill map.

Next: Config and secrets

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Who owns whom, and why undo is instant

You write the wiring for the search service; Lina reviews. The selector, the labels and the ports must line up across three objects.

Worked example

You wire search-api end to end

Move search-api into the search namespace so other services can call it by name, guests reach it at api.stayhub.com/search, and a typo can't silently cut it off.

1Pick the labels once
Define app: search-api in one place (a chart value) and use it in the Deployment's selector, its pod template and the Service's selector.
Why: Thursday's outage was a label drifting in one of three places. One source of truth removes that class of bug.
2Name the container port
In the pod spec: ports: [{ name: http, containerPort: 8080 }]. In the Service: port: 80, targetPort: http.
Why: A named port follows the container if it ever changes to 8081, instead of silently forwarding to a port nothing listens on.
3Keep it internal
Service type ClusterIP (the default). Callers inside the cluster use search-api.search or the full search-api.search.svc.cluster.local.
Why: Lina: "nothing internal gets a public load balancer. The fewer front doors, the fewer to secure and pay for."
4Route outside traffic through the Ingress
Add a rule to the shared Ingress: host api.stayhub.com, path /search → Service search-api, port 80. Check the Ingress shows an ADDRESS.
yaml
```
rules:
  - host: api.stayhub.com
    http:
      paths:
        - path: /search
          pathType: Prefix
          backend: { service: { name: search-api, port: { number: 80 } } }
```
Why: An Ingress is only a request; the controller does the routing. No ADDRESS means no controller picked it up.
5Verify the chain
get endpointslices (are there endpoints?), then a curl from a throwaway pod to http://search-api.search/ready, then from outside to https://api.stayhub.com/search.
Why: Walk the chain from the inside out: endpoints, Service, Ingress. Each check rules out one layer in seconds.

Service type	Reachable from	Cost / side effects	Use for
ClusterIP	Inside the cluster only	None	Service-to-service calls; anything behind an Ingress
NodePort	Any node IP on port 30000–32767	Opens a port on every node	Bare-metal setups, or as the hop under an external LB
LoadBalancer	A cloud LB IP (public or internal)	One cloud load balancer per Service, billed monthly	The ingress controller itself; non-HTTP (TCP/UDP) services
Headless (clusterIP: None)	DNS returns pod IPs directly	No load balancing by Kubernetes	Databases, or clients that balance across pods themselves

Default to ClusterIP and add exposure deliberately.

bash

# 1. Does the Service select any pods? Empty = selector or readiness problem.
kubectl -n bookings get endpointslices -l kubernetes.io/service-name=booking-api
kubectl -n bookings get pods -l app=booking-api --show-labels
# 2. Who owns what? (Deployment -> ReplicaSet -> Pod)
kubectl -n bookings get rs -l app=booking-api
# 3. Is the Service reachable from inside the cluster?
kubectl -n bookings run tmp --rm -it --image=curlimages/curl -- \
  curl -s http://booking-api.bookings.svc.cluster.local/ready
# 4. Did an ingress controller pick up the Ingress?
kubectl get ingressclass
kubectl -n bookings describe ingress public-api

Walk the chain when traffic doesn't arrive.

Interactive simulation

Visual diagram

Guests

Cloud LB (1 for all)

Ingress controller

Service booking-api (ClusterIP)

Deployment booking-api

ReplicaSet -5c4b (v3.3.0)

ReplicaSet -7d9f (v3.2.0)

Pod -5c4b-a

Pod -5c4b-b

Pod -5c4b-c

Click a component

Ownership (top) and traffic (left to right) for the booking API in the `bookings` namespace.

Analogy

A hotel chain's staffKubernetes

A hotel chain's staffBadges: "receptionist, Amsterdam"KubernetesLabels on pods
A hotel chain's staffHead office: "Amsterdam needs 6"KubernetesA Deployment
A hotel chain's staffA shift roster per training versionKubernetesA ReplicaSet per app version
A hotel chain's staffFront-desk number rings badge-wearers on dutyKubernetesA Service routing to ready pods
A hotel chain's staffThe switchboard routing calls by topicKubernetesAn Ingress routing by path
A hotel chain's staffDepartments with their own budgetsKubernetesNamespaces

Where the analogy breaks: a receptionist with a misspelled badge would notice nobody calls. Kubernetes doesn't notice. The Service quietly has zero pods behind it.

Check your understanding

Q1/3Decision

Match each need to the Service type you'd choose:

Pricing service called only by the booking API

The ingress controller in a cloud cluster

A partner TCP feed on port 7000 in a cloud cluster

A bare-metal cluster behind an existing hardware load balancer

How confident are you?

Practice ladder

Step 1 · Trace

Trace a call to reviews-api

Scenario Warm-up

In namespace reviews:

# Deployment pod template
labels: { app: reviews-api, tier: api }
ports: [{ name: http, containerPort: 4000 }]
# Service reviews-api
selector: { app: reviews-api }
ports: [{ port: 80, targetPort: 3000 }]

There are 4 pods: 3 Ready, 1 still starting. The web app calls http://reviews-api.reviews/reviews?hotel=42.

Trace the call: which pods are in the endpoints, which port the traffic reaches, and what the web app sees.

Step 2 · Complete

Put Lina's debugging steps in order

Debugging Practice

Guests get 503 at https://api.stayhub.com/pricing. Lina's checklist is shuffled. Order the steps from the inside out and say what each rules out:

A. kubectl -n pricing describe ingress public-api: does it have an ADDRESS and the right backend?
B. kubectl -n pricing get endpointslices -l kubernetes.io/service-name=pricing-api
C. curl from a throwaway pod to http://pricing-api.pricing/ready
D. kubectl -n pricing get pods -l app=pricing-api: are pods Running and Ready?
E. Test from outside: curl -i https://api.stayhub.com/pricing

Step 3 · Solve

Partners can't reach the new availability API

Debugging Practice

You expose the new availability-api to hotel partners at https://partners.stayhub.com/availability. You created a Deployment (3 pods, Ready), a ClusterIP Service and this Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: partners, namespace: availability }
spec:
  rules:
    - host: partners.stayhub.com
      http:
        paths:
          - path: /availability
            pathType: Prefix
            backend: { service: { name: availability-api, port: { number: 80 } } }

kubectl get ingress shows the ADDRESS column empty. In-cluster curls to the Service work. Find the cause and the fix, and say what else partners need before it works over HTTPS.

Step 4 · Decide

Separate cluster for staging?

Decision Stretch

After the load-test incident, two proposals:

A (Omar): a separate cluster for staging. About €900 a month more, and a second cluster to upgrade.
B (Karim): keep staging in the production cluster, and add default-deny network policies plus RBAC per namespace. Free, about 3 days of work.

Facts: staging is used by 25 engineers; it runs load tests monthly; a cluster upgrade takes Lina about 2 days per quarter; StayHub handles card payments.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

A request arrives at api.stayhub.com/bookings. Explain every object it passes through until it reaches a container, and how each object finds the next one.

0 words