Episode 58 · The fix that undid itself

Infrastructure as code

Saturday, 19:30: at peak, checkout starts failing with "too many connections", the exact error Omar fixed by hand two weeks ago. Nobody touched that setting. Terraform did, last night.

Instead of clicking around a cloud console, you describe your servers and networks in files, and a tool builds exactly that.

30 min intermediateDeclarative IaC & stateTerraform plan / apply workflowInfrastructure driftSecrets managementGitOpsDesired state & reconciliation

Problem
A booking platform needs identical staging and production environments, a second region for disaster recovery, and a record of who changed which firewall rule and why.
Why the simple system fails
Clicking resources together in a cloud console is unrepeatable, unreviewed and undocumented. Environments drift apart, nobody knows what exists, and rebuilding after a disaster takes days of guesswork.
Engineering concept
Declare the desired infrastructure in code; a tool compares it with what exists and makes the minimum changes. Changes go through Git, review and a pipeline, like application code.
Technology
Terraform/OpenTofu (HCL), remote state with locking, plan/apply, modules, a secrets manager, GitOps reconcilers.
Trade-offs
IaC adds a learning curve, state to protect and slower 'quick fixes', in exchange for repeatability, review, auditability and fast recovery.

The problem

2 min

Saturday, 19:30. Peak hour.

Omar · On-call SRE
Checkout failing: "too many connections". max_connections is 400. Two weeks ago I raised it to 800 in the console.
Omar · On-call SRE
Only change since: Friday 18:00, Terraform apply for the read-replica PR.
You · Engineer at StayHub
But that PR was only about the replica.

The big idea

4 min

Code is the truth, even when it's wrong

The big idea

The code is the source of truth. A hand fix isn't done until it's in the code.

Peak needed ~620 connections, 400 were allowed: 31 minutes, ~560 failed checkouts. Reverting drift is a feature, not a bug: it keeps the code honest. The mistake was leaving the fix in the console.

How it works

5 min

The life of an infrastructure change

1
Describe what should exist
: infrastructure described in files kept in Git. Declarative tools like take "this should exist" and work out the steps, so they can always compare.
2
State
The is Terraform's record of which real resource belongs to which line of code. Stored remotely, locked, protected.
3
Plan
The lists every change needed to match the code: + create, ~ change in place, -/+ destroy and recreate, - destroy.
4
Review the plan, not just the diff
Friday's reviewer looked for the replica and skimmed past line 23: ~ value = "800" -> "400". -/+ or - on anything holding data means stop.
5
Apply
CI applies exactly the reviewed plan, no more.
6
Drift
is a difference between code and reality, usually a hand change. The next apply removes it, whatever that apply was for.

Watch a decision being made

4 min

Worked example

You add the guardrails; Lina checks them

A hand fix must never be silently undone again, and dangerous plan lines must get noticed, without slowing normal changes.

1Codify the fix now, with its reason
max_connections = "800", with a comment linking the incident.
hcl
```
parameter {
  # Raised from 400 after the Saturday-peak incident:
  # peak needs ~620 connections; PgBouncer is next.
  name  = "max_connections"
  value = "800"
}
```
Why: Until it's in code, the next apply undoes it. The comment stops someone "tidying" it back.
2Detect drift every night
At 03:00, terraform plan against production; any non-empty plan goes to on-call.
Why: Drift found next morning is a 5-minute PR; found by an unrelated apply, it's an outage.
3Make dangerous lines loud
A bot summarises each plan and flags in red any change to a database, its parameters, or any -/+/- on stateful resources.
Why: A 41-line plan hides line 23; a summary at the top doesn't.
4Close the side door
Console read-only; a logged break-glass role for emergencies, requiring a codify PR within 24 hours.
Why: Emergencies need a fast path, but every fast path must lead back into code.

In real life

3 min

A config language versus a programming language

Terraform / OpenTofu: HCL

HashiCorp's Terraform and the open-source fork OpenTofu describe infrastructure in HCL, with a plan before every apply
Limited logic keeps files readable
Huge ecosystem of providers

Fits most teams: plans easy to review.

Pulumi / AWS CDK: real languages

Publicly offer infrastructure in TypeScript, Python and others
Loops, types, tests, abstractions
Easy to build clever layers that hide what gets created

Fits heavy abstraction, with discipline about readability.

What goes wrong

3 min

Hot-fixing in the console and never codifying

The next apply silently reverts it.

Instead

Codify within 24 hours; nightly drift detection.

Approving the diff without reading the plan

A rename can mean "destroy the database".

Instead

A plan summary on every PR; a policy that blocks destroying stateful resources.

Protecting only production

Staging data is lost just as easily.

Instead

Deletion protection on every environment's databases.

Trade-offs

3 min

Approach	Fits	You pay
Click-ops (console)	A sandbox experiment	Not repeatable; environments drift
Declarative IaC (Terraform)	Any long-lived environment	State to protect; plans must be read
IaC in a programming language	Heavy abstraction needs	Harder to predict what gets created
reconciliation	Kubernetes config, many teams	Hand fixes reverted within minutes

Monday, 10:00. The drift postmortem.

Karim · CTO
Read-only console? On a Saturday night I want Omar to fix it in 30 seconds, not open a PR.
Omar · On-call SRE
My 30-second fix is what took us down. Give me a break-glass role: fix in 30 seconds, PR within 24 hours, tracked.
Karim · CTO
More process during incidents.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Who may run terraform apply: only CI after a reviewed plan, never the agent from your laptop.
Which resources are protected from destruction: the bookings database, replicas, the photo bucket.
Where secrets live: in the , referenced by name, never in .tf files.

Tell Claude Code

Codify the bookings database connection limit Omar raised by hand at the Saturday peak: max_connections must be 800 in our production Terraform, with a comment explaining the incident and date. Staging keeps its value. Change nothing else; do not run terraform apply. Run terraform plan for production and show me the full summary. I expect exactly one in-place change to the parameter group, nothing added or destroyed. If the plan shows anything else, especially a replace on a database, stop and explain.

Check before you accept

The summary reads "0 to add, 1 to change, 0 to destroy".
No -/+ or must be replaced on databases, buckets or volumes.
No unexpected ~ lines: they're drift that the apply would revert.

Red flags

It runs terraform apply or -auto-approve itself.
It removes prevent_destroy to let a plan through.
It edits the state file by hand or deletes the lock.

Check yourself

2 min

Q1/3Reasoning

Your Terraform code declares 3 app servers. Currently 5 exist (all managed by this configuration). What does terraform apply do?

How confident are you?

Next episode: a deploy at 18:00 on a Friday replaces every booking server at once, and checkout is down until 18:09.

Finished?

Mark it complete to update your skill map.

Next: Deployment strategies: recreate, rolling, blue-green, canary, shadow

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Guardrails around plan and apply

Worked example

You add the guardrails; Lina checks them

Make sure a hand fix can't be silently undone again, and that dangerous plan lines get noticed, without slowing down normal infrastructure changes.

1Codify the fix now
PR: max_connections = "800" in the parameter group, with a comment linking the incident and the plan to move the newer services behind the connection pooler.
Why: Until it's in code, the next apply will undo it again. The comment stops someone "tidying" it back to 400.
2Detect drift every night
A scheduled job runs terraform plan against production at 03:00 and posts any non-empty plan to the on-call channel.
Why: Drift found the next morning is a 5-minute codify PR. Drift found by an unrelated apply two weeks later is an outage.
3Make dangerous lines loud
A bot posts a plan summary on each PR: counts of add/change/destroy, and a red warning for any change to a database, its parameters, or any -/+/- on stateful resources.
Why: Reviewers read what's put in front of them. A 41-line plan hides line 23; a summary at the top doesn't.
4Close the side door
Console access becomes read-only for managed resources. A break-glass role exists for emergencies; using it opens a ticket that requires a codify PR within 24 hours.
Why: Emergencies still need a fast path, but every fast path must lead back into code. Lina's check: the break-glass role must also be logged and alerted.
5Guard stateful resources
deletion_protection = true and prevent_destroy on databases and buckets, plus a policy check that fails any plan destroying them.
Why: Some mistakes are too expensive to rely on a human noticing them in review.

hcl

terraform {
  backend "s3" {                       # remote, versioned, encrypted state
    bucket       = "stayhub-tf-state"
    key          = "booking/production.tfstate"
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true                # no two applies at once
  }
}

module "network" {                     # reusable module: VPC + subnets in 3 zones
  source = "../../modules/network"
  name   = "booking-production"
}

resource "aws_db_instance" "bookings_primary" {
  instance_class              = var.db_size
  multi_az                    = true
  parameter_group_name        = aws_db_parameter_group.bookings.name
  manage_master_user_password = true   # generated + stored in the secrets manager
  deletion_protection         = true
  lifecycle { prevent_destroy = true }
}

Remote locked state, a module, a secret that never appears in code, and two guards against deleting the database.

Visual diagram

Engineer

Git repo + PR

Pipeline: plan / apply

Remote state + lock

Secrets manager

Cloud provider API

Bookings DB

Network + LB

Click a component

An infrastructure change flowing through Git, review and a pipeline, with remote state and a secrets manager.

Analogy

Constructing a buildingInfrastructure as code

Constructing a buildingThe architect's blueprintInfrastructure as codeYour Terraform files
Constructing a buildingThe manager's list of what's installedInfrastructure as codeTerraform state
Constructing a buildingThe contractor's quote before workInfrastructure as codeterraform plan
Constructing a buildingBuilding exactly what was quotedInfrastructure as codeterraform apply
Constructing a buildingA tenant knocking down a wall secretlyInfrastructure as codeDrift
Constructing a buildingMaster keys kept in a safeInfrastructure as codeA secrets manager

Where the analogy breaks: rebuilding a building takes months. With IaC, you can tear down and recreate a whole environment in minutes, which is exactly what makes it so powerful, and so dangerous if you skip reviewing the plan.

Check your understanding

Q1/3Reasoning

Marking a Terraform variable as sensitive means the secret is never stored anywhere readable, so it's safe to keep database passwords that way without protecting the state.

How confident are you?

Practice ladder

Step 1 · Trace

Read this plan

Debugging Warm-up

A PR titled "Bigger instances for search" produces this plan for production:

~ aws_autoscaling_group.search   max_size: 6 -> 10
-/+ aws_db_instance.search_db    (forces replacement: storage_type "gp2" -> "io2")
~ aws_security_group_rule.db_in  cidr_blocks: ["0.0.0.0/0"] -> ["10.0.0.0/16"]
+ aws_cloudwatch_metric_alarm.search_cpu
Plan: 1 to add, 2 to change, 1 to destroy.

For each line, say what will happen and whether you'd approve. What do you do overall?

Step 2 · Complete

Finish the environment files

Code Practice

Staging and production call the same booking module. Fill in the gaps in the production file so production differs from staging only where it should, and explain each value.

# envs/staging/main.tf
module "booking" {
  source              = "../../modules/booking"
  environment         = "staging"
  db_size             = "db.t4g.medium"
  multi_az            = false
  deletion_protection = true
  app_min_instances   = 1
}

# envs/production/main.tf
module "booking" {
  source              = "../../modules/booking"
  environment         = "production"
  db_size             = ___
  multi_az            = ___
  deletion_protection = ___
  app_min_instances   = ___
}

Step 3 · Solve

Bring the hand-built Singapore resources under code

Design Practice

When Singapore opened, about 150 resources were created by hand in the console (network, load balancer, app VMs, a Redis cache, DNS records). Production traffic runs there now. You need it in Terraform within 6 weeks, with no downtime, so it can be rebuilt in another region if needed.

Plan the work: in what order, how you prove the code matches reality, and how you stop new hand changes during and after.

Step 4 · Decide

Apply from laptops?

Decision Stretch

Two senior engineers ask to keep running terraform apply from their laptops "for speed" when CI is slow (an infra pipeline run takes about 12 minutes). Everyone else applies through CI.

Facts: last quarter, one laptop apply used an out-of-date local branch and reverted another team's merged change; laptop credentials have broad admin rights; CI posts plans on PRs and applies only reviewed plans.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to an engineer who builds everything in the cloud console why the team uses Terraform, and how a change gets from their idea to a real resource.

0 words