Episode 67 · The key in the repo

Config and secrets

Wednesday, 10:15: the payment provider wants StayHub's live key rotated by 18:00. You find it in git, in a Kubernetes Secret, base64-encoded and readable by 140 people.

Build one image, hand it different settings in each environment, and learn why a Kubernetes "Secret" isn't as secret as its name.

30 min advancedConfig separated from code (12-factor)Env vars vs mounted files for configurationSecrets managementConfigMaps & SecretsLeast privilege for agents

Problem
The same booking-api image must run in dev, staging and production with different database URLs, feature flags and a production Stripe key that only production may see.
Why the simple system fails
Baking config into the image means a rebuild per environment; committing secrets to Git leaks them to everyone with repo access, forever; a Kubernetes Secret alone is readable by anyone who can read Secrets or create pods in the namespace.
Engineering concept
Separate config from code (12-factor), inject it at runtime as env vars or files, treat secrets as a separate class with encryption, least-privilege access, auditing and rotation.
Technology
ConfigMaps, Secrets, etcd encryption at rest (KMS), RBAC, External Secrets Operator / CSI driver, cloud secret managers or Vault, workload identity.
Trade-offs
Plain Kubernetes Secrets are simple but weakly protected; external managers add auditing and rotation at the cost of another critical dependency and more moving parts.

The problem

2 min

Wednesday, 10:15. An email from the payment provider.

Karim · CTO
The provider wants our live payment key rotated by 18:00. A former contractor's laptop with repo access was stolen.
You · Engineer at StayHub
It's in deploy/bookings/payments-secret.yaml. It's a Kubernetes Secret, so it's encoded at least.
Lina · Senior engineer, your mentor
You lead this. First: who can read that key today?

The big idea

4 min

Encoded is not encrypted

The big idea

Encoded isn't encrypted. A secret is protected by who can read it, and by being easy to change, not by how it's written.

Deleting the file doesn't help: git history keeps it in every clone, the stolen laptop included. Only rotating the key makes the leaked value useless. Then every value needs a proper home, so this can't happen again.

How it works

5 min

Three kinds of values, three homes

1
Code: in the image
Built once, promoted unchanged from staging to production. No settings baked in, so the image you tested is the one that ships.
2
Settings: a ConfigMap
A ConfigMap holds plain settings like SEARCH_TIMEOUT_MS=800. Different per environment, fine to keep in git.
3
Secrets: a secret manager
A secret manager is a cloud service that stores secrets encrypted, logs every read and supports rotation (replacing a value without downtime). For the payment key, database password, TLS keys.
4
Synced into the cluster
An operator (External Secrets) copies each one into a Kubernetes every hour. Git holds only a reference, never a value.
5
Injected at start
Env vars are read once, at process start. Mounted files refresh in place, so a rotated secret arrives without a restart if the app re-reads it (once a minute).
6
Who can read it
Turn on encryption at rest for and limit get secrets with . Surprise: anyone who can create pods in a namespace can mount, and read, its Secrets.

Watch a decision being made

4 min

Worked example

You rotate the payment key before 18:00

Replace the leaked live key before 18:00 without failing a single payment, and make sure the new key never lands in git. You lead; Lina reviews each step.

1Find every reader, and how it reads
payments-api (6 pods, env var) and the refunds worker (2 pods, env var).
Why: You can't switch readers you don't know about. Env var or file decides whether pods need a restart.
2Create the new key alongside the old
A second live key in the provider's dashboard; both work for now. Store it in the secret manager at prod/payments/key, and replace the committed manifest with an ExternalSecret pointing there, mounted as a file.
Why: The overlap means no moment where a pod holds a key that doesn't work.
3Roll the pods
kubectl rollout restart deploy/payments-api deploy/refunds-worker, and watch them become ready.
Why: Running pods still hold the old key in memory. A rolling restart replaces them a few at a time, without downtime.
4Verify, then revoke
The provider's dashboard shows zero calls on the old key for 15 minutes, and payments succeed. Only then revoke it.
Why: "Revoke last. The dashboard tells you if anything still uses the old key, which a checklist can't."

In real life

3 min

Built-in Secrets, or an external manager?

Plain Kubernetes Secrets

Built in: nothing else to run, works with every chart
Base64 in etcd unless encryption at rest is turned on
No log of who read what, no rotation workflow

Fine for low-risk values with encryption on and strict access.

An external manager (HashiCorp Vault, AWS Secrets Manager…)

Widely used: secrets stored encrypted outside the cluster
Every read audited; versions and scheduled rotation built in
One more critical service: if it's down, new pods can't get secrets

Worth it for high-risk secrets and anything that must rotate.

What goes wrong

3 min

Rotating by replacing the value in place

Pods still holding the old value fail as soon as it's revoked.

Instead

Overlap: add new, roll pods, verify, then revoke old.

Committing a Secret manifest because "it's encoded"

Base64 decodes instantly, and git keeps it forever, in every clone.

Instead

Commit references (ExternalSecret) or encrypted files. If a secret was committed, rotate it.

Logging or exposing the environment

Crash reporters and debug endpoints copy secrets into systems with far wider access.

Instead

Mount secrets as files; never return env in responses or logs.

Trade-offs

3 min

Who reads prod Secrets	Gains	Costs
All engineers, standing access	Fast debugging, no process	25 leak sources, no record of who read what
Pipelines + a break-glass role	Few readers, every read logged	Minutes of friction in incidents, setup work

Thursday, 11:00. Deciding who may read production Secrets.

Karim · CTO
Engineers debug production every week. Without Secrets, a 10-minute fix becomes an hour of waiting for Omar.
Omar · On-call SRE
If 25 people can read the payment key, we're back to yesterday with extra steps.
Karim · CTO
We hire adults. Every request to you is a delay during an incident.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Where secrets live: the payment key in the cloud secret manager, never in the deploy repo.
Who and what may read each secret, including who can create pods in bookings (the same as reading its Secrets).
The rotation procedure: overlap first (new key, roll pods, verify), revoke the old key last.

Tell Claude Code

Move the payment key out of deploy/bookings/payments-secret.yaml. Create an ExternalSecret reading prod/payments/live-key from our secret manager (1-hour refresh), mounted into payments-api as a file at /etc/secrets/payment-key instead of the PAYMENT_KEY env var; re-read it at most once a minute. Never print the key: not in output, logs, commits or tests. Don't revoke anything. Plan first, then write a rotation runbook for 6 pods: new key, pods pick it up, verify on every pod, then revoke the old.

Check before you accept

Search the diff and Claude's output: the key value appears nowhere.
The repo holds only a reference, and Claude reminds you the old key is in history and must be rotated.
In the runbook, revoke is the last step, after all 6 pods use the new key.

Red flags

It calls a Kubernetes Secret "encrypted" because it is base64.
It prints or logs the secret to "verify it works".
It "removes" the leak by deleting the file and doesn't mention rotation.

Check yourself

2 min

Q1/3Reasoning

Because Secret values are base64-encoded, committing a Secret manifest to a private Git repo is acceptable for production credentials.

How confident are you?

Next episode: every deploy of booking-api costs about 2% of requests for 40 seconds, and nobody can say why.

Finished?

Mark it complete to update your skill map.

Next: Health checks and rollouts

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Rotate the key without failing a payment

You lead the rotation; Lina reviews each step. The deadline is 18:00, and no payment may fail.

Worked example

You rotate the payment key before 18:00

The live payment key leaked through the deploy repo. Replace it before 18:00 without failing a single payment, and make sure the new key never lands in git.

1Find every reader
Search the cluster and repos for the key name: payments-api (6 pods, env var), the refunds worker (2 pods, env var). Note how each reads it.
Why: You can't switch readers you don't know about. And env var vs file decides whether pods need a restart.
2Create the new key alongside the old
In the provider's dashboard, create a second live key. Both work for now. Store the new one in the secret manager at prod/payments/key.
Why: The overlap is what makes the rotation safe: there's no moment when a pod holds a key that doesn't work.
3Point the cluster at the manager
Replace the committed Secret manifest with an ExternalSecret that references prod/payments/key, and mount the resulting Secret as a file in both apps.
yaml
```
volumeMounts:
  - { name: payments, mountPath: /etc/secrets, readOnly: true }
volumes:
  - name: payments
    secret: { secretName: payments-secret, defaultMode: 0400 }
```
Why: Git now holds a pointer, not a value. Files refresh in place, so future rotations won't need restarts.
4Roll the pods
This release also switches the apps from env var to file, so roll both Deployments: kubectl rollout restart deploy/payments-api deploy/refunds-worker, and watch them become ready.
Why: Pods started before the change still hold the old key in memory. A rolling restart replaces them a few at a time, without downtime.
5Verify, then revoke
Check the provider's dashboard: the old key has had zero calls for 15 minutes, and payments succeed. Only then revoke the old key. Rewrite nothing in git history; the old key is now useless.
Why: Lina: "revoke last. The dashboard tells you if anything still uses the old key, which a checklist can't."

	Env vars	Mounted files
Updates after change	Only on pod restart	Refreshed in place (not for `subPath` mounts)
App support	Universal; every framework reads env	App must read (and re-read) files
Leak risk	Higher: inherited by child processes, dumped by crash reporters and debug tools	Lower: only readable through the file, with file permissions
Good for	Plain config, values that change only with a deploy	Secrets, certificates, rotated credentials

Many teams use env vars for config and files for secrets.

Visual diagram

Git repo (manifests)

Secret manager

API server

External Secrets Operator

etcd

booking-api pod

Click a component

Where config and secrets live, and how they reach the booking-api container.

Analogy

A hotel chainConfig and secrets

A hotel chainThe standard room, same in every hotelConfig and secretsThe image
A hotel chainThe welcome sheet: Wi-Fi, breakfast hoursConfig and secretsA ConfigMap
A hotel chainA master key in an envelope marked SECRETConfig and secretsA Kubernetes Secret (base64)
A hotel chainLocking the office safe at nightConfig and secretsEncryption of stored data
A hotel chainA key office with a logbookConfig and secretsA secret manager

Where the analogy breaks: a locked safe stops thieves, but not staff who know the combination. Encryption at rest is the same. You still need to control who is allowed to read.

Check your understanding

Q1/3Scenario

In the production bookings namespace, which of these permissions effectively let someone read the Stripe key stored in a Secret? Select all that apply.

Select all that apply.

How confident are you?

Practice ladder

Step 1 · Trace

Trace a timeout change

Scenario Warm-up

search-api has 4 pods. They load settings with envFrom: configMapRef: { name: search-config }. At 10:00 you change SEARCH_TIMEOUT_MS from 800 to 1500 in search-config.

At 10:20 one pod is evicted and recreated.
At 11:00 nothing else has happened.

Trace: at 11:00, which timeout does each pod use? What do guests experience? What would have made this change predictable?

Step 2 · Complete

Order the database password rotation

Scenario Practice

The production database password for booking-api (12 pods, reads it from a mounted file, re-reads every minute) must be rotated. Lina's steps are shuffled. Put them in order and fill in the missing step X.

A. Revoke (drop) the old password in the database
B. Store the new password in the secret manager
C. Create a second database user/password that works alongside the old one
D. Wait for the sync and the file refresh; check that new connections use the new user
X. ___

Step 3 · Solve

Who can read the key?

Debugging Practice

After the rotation, an audit asks who can read payments-secret in namespace bookings. These roles exist:

developers: get, list on pods and logs; create on pods (to run debug pods).
ci-deployer: create, update on Deployments.
support: get on ConfigMaps.
sre: get, list on Secrets.

Which roles can effectively read the payment key, and how? Propose changes.

Step 4 · Decide

Feature flags: ConfigMap or flag service?

Decision Stretch

Maya wants to turn the new checkout on for 10% of guests, then 50%, then 100%, and switch it off within a minute if conversions drop. Today checkout's flags are still old-style env vars from a ConfigMap; changing one means a rollout of 12 pods (about 6 minutes).

Options: - A: keep the ConfigMap, but mount it as a file and re-read every 30 s. - B: move the flag to the hosted feature-flag service StayHub adopted for the cancellation flow (DevOps module): percentage rollouts, local evaluation in the SDK, about a day of work.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

A teammate says: "Our Stripe key is safe, it's in a Kubernetes Secret." Explain what that does and doesn't protect against, and what you'd add.

0 words