Incident response: alerts, on-call, mitigation and blameless postmortems
Saturday, 20:12, peak booking hour: checkout errors jump to 18%, five engineers join the call and start changing things at once, and Karim asks who's in charge. Tonight, you are.
Something is broken and users are waiting. Here's how good teams stay calm, fix it fast, and learn from it.
- Problem
Production breaks at the worst moment. Customers can't book, five engineers jump into a chat, and nobody knows who is doing what.
- Why the simple system fails
Alerts on CPU and disk fire all night but miss the real outage; everyone debugs, nobody communicates; the team hunts the root cause for an hour while a rollback would have fixed it in five minutes; the postmortem blames a person and nothing changes.
- Engineering concept
Alert on user-facing symptoms, route to a prepared on-call engineer with a runbook, run the incident with explicit roles, mitigate first, then learn without blame.
- Technology
SLO burn-rate alerts, paging tools (PagerDuty, Opsgenie and similar), runbooks, an incident channel and status page, rollbacks and feature flags, postmortem templates with tracked action items.
- Trade-offs
More alerts catch more problems but burn out on-call; fast mitigation (rollback) may hide evidence; formal process helps big incidents but is overhead for small ones.
The problem
2 min- Omar · On-call SREPage at 20:09: checkout errors 18%, p99 6 s. Database CPU 95%.
- Karim · CTOThree of you are changing things at once. Who's in charge?
- Lina · Senior engineer, your mentorYou are. You lead this one; I'll only check your calls.
The big idea
4 minDuring an incident, restore service first; finding the root cause is the postmortem's job.
Think of a fire brigade, not a detective agency. The chief on site doesn't hold a hose; she decides where the hoses go. Nobody investigates the faulty wiring while the building burns.
How it works
5 min- 1Detect from a symptom
20:09: the checkout burn-rate alert pages Omar. Page on what guests feel (errors, latency), never on causes like CPU. Burn rate 18% ÷ 0.1% = 180: the month's is gone in ~4 hours.
- 2Declare and assign roles
20:12: SEV1. The (you) coordinates and decides, but never types. Omar is ops lead, the only one changing systems. Maya handles comms. Someone writes the timeline.
- 3Ask what changed
Deploys, flags, config, traffic. Most incidents follow a change. Pricing v4.12 went out at 19:58.
- 4Mitigate, reversibly
Mitigation is anything that restores service before the cause is understood: roll back, flip a flag, fail over. A runbook, linked from the alert, lists the safe ones in order. 20:21: roll back pricing. Announce it, change one thing, watch the dashboard.
- 5Communicate on a clock
Status page and support at 20:16, 20:30, 20:40: what's broken, for whom, since when, next update time. No guesses about the cause.
- 6Close, then learn
0.1% errors at 20:26, closed at 20:40: ~21 minutes and ~2,800 failed checkouts. The missing index goes into a blameless on Monday: how did the system allow it, not who did it.
Watch a decision being made
4 minYou run the SEV1, Lina checks
Checkout errors are at 18% at Saturday peak. Restore service, keep people informed, and set up the learning, in that order.
- 1Size the impact
RED dashboard: every checkout route fails, search is fine. Burn rate 180: the 30-day budget is gone in ~4 hours. That's a SEV1.
Why: Lina: "Scope and speed decide the severity, and the severity decides who you wake."
- 2Declare and assign roles
"SEV1. I'm IC. Omar is ops lead and the only one changing systems. Maya, comms. Sam, timeline."
Why: Five people changing things at once is how the 20:15 scale-up happened. One operator means every change is announced and traceable.
- 3Pick the safest reversible mitigation
Deploy log: v4.12 at 19:58. Traces: the slow span is the new
price_historyquery. Follow the runbook's order: roll back first.yamlfirst_actions: # safe, reversible, in this order - "Deployed in the last 60 min? -> roll back" - "Feature flag changed? -> revert the flag" - "DB CPU > 90%? -> disable 'price-history' widget (flag)"Why: Reversible actions that match the evidence are low risk. Restarts and scaling are reflexes that often add load.
- 4Confirm, close, hand over
Errors 0.1% at 20:26; watch 15 minutes; close at 20:40. Keep the slow-query logs and traces; schedule the postmortem for Monday.
Why: Closing too early risks a second wave. Evidence captured now keeps the postmortem factual.
In real life
3 minEtsy: blameless reviews, for the team
- Its engineering blog publicly described "blameless postmortems" back in 2012
- People explain what they did and why it made sense at the time, without fear of punishment
- The goal is learning inside the company
Optimised for honesty and internal improvement.
Cloudflare: detailed public postmortems
- Publishes detailed write-ups of major outages on its public blog, often within days
- Explains timelines and technical causes to customers
Optimised for customer trust; needs care about what's safe to publish.
What goes wrong
3 minIt sends more concurrent slow queries to a database already at 95%.
Take load off the bottleneck: roll back, or switch off the expensive feature with its flag.
Changes collide, the timeline is lost, nobody updates support.
One incident commander, one ops lead who announces each change, one comms lead on a cadence.
People hide mistakes, and the real gaps (no load test, no canary) stay.
Blameless reviews with owned, dated action items that change the system.
Trade-offs
3 min| Option | Choose when | You pay |
|---|---|---|
| Roll back | A deploy, flag or config change lines up with the symptoms | May remove a wanted feature; impossible after some migrations |
| Fix forward | Rollback is unsafe, and the fix is small and understood | Code, review, build, deploy during the outage; risk of a second bad change |
| Mitigate around (flag, failover) | The cause is outside your release, or unknown | A degraded experience; the switches must exist beforehand |
- Maya · Product managerv4.12 also has the weekend promotion marketing announced at 18:00. Fix forward: just add the index.
- Omar · On-call SREBuilding that index under this load takes 20+ minutes and adds load. A rollback takes 3.
- Maya · Product managerAnd then we explain why the promotion vanished on its first night.
Your job as tech lead
4 minClaude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.
- Who is : a human. Claude Code can read logs and draft commands; you decide what changes in production, one change at a time.
- Which commands the agent may run during an incident: read-only by default. Rollbacks, scaling and database changes need your explicit yes.
- What the changes: system fixes (an index check in CI, a load test), each with an owner and a date. No blame.
SEV1: checkout errors 18% since about 20:05, database CPU 95%. I'm incident commander. Read only: do not deploy, scale, restart or change anything. Within five minutes, list every deploy and config change since 19:30, check whether the slow queries started after one of them, and give me the exact rollback command for the likeliest change with what it would undo. If the evidence points nowhere, say so. I will run the rollback myself.
- Claude only ran read-only commands: scroll its tool calls for any deploy, scale or restart.
- The timing holds: v4.12 at 19:58, errors from ~20:05. A cause that starts after the symptom is not the cause.
- After the rollback, watch the error rate, not Claude's opinion: back to ~0.1% within minutes.
- It suggests scaling app servers to "absorb load" while the database is the bottleneck.
- It runs a fix (creating an index, restarting the database) without asking, mid-incident.
- Its postmortem names a person as the root cause, or ends with "be more careful".
Check yourself
2 minNext episode: a security researcher changes one number in a URL and sees another hotel's guest.
Mark it complete to update your skill map.
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
Technical deep dive
You run the SEV1, Lina checks
Checkout errors are at 18% at Saturday peak. Restore service, keep people informed, and set up the learning, in that order.
- 1Size the impact
RED dashboard: every checkout route fails, search is fine. Burn rate 18% ÷ 0.1% = 180: the 30-day budget is gone in ~4 hours.
Why: Lina's check: "Good. Scope and speed decide the severity, and the severity decides who you wake."
- 2Declare and assign roles
"This is a SEV1. I'm IC. Omar is ops lead and the only one changing systems. Maya, comms. Sam, timeline."
Why: Five people changing things at once is how the scale-up happened. One person changing systems means every change is announced and traceable.
- 3Ask what changed
Deploy log: pricing v4.12 at 19:58. Flags: none. Traffic: normal Saturday. Traces: the slow span is the new
price_historyquery.Why: Most incidents follow a change. The deploy log is faster than any debugger.
- 4Pick the safest reversible mitigation
Follow the runbook's order: roll back first. Scale-up is not on the list for a database bottleneck.
yamlfirst_actions: # safe, reversible, in this order - "Deployed in the last 60 min? -> roll back" - "Feature flag changed? -> revert the flag" - "Payment provider slow? -> enable backup provider (flag)" - "DB CPU > 90%? -> disable 'price-history' widget (flag) to shed load" escalate: "Declare SEV1 if checkouts failing > 5% for 10 min"Why: Reversible actions that match the evidence are low risk. Restarts and scaling are reflexes that often add load.
- 5Communicate on a clock
Maya posts at 20:16, 20:30 and 20:40: what's broken, for whom, since when, next update time.
Why: Silence makes support, hotels and guests assume the worst; a cadence stops people interrupting the ops lead for news.
- 6Confirm, close, hand over
Errors 0.1% at 20:26; watch 15 minutes; close at 20:40. Keep the slow-query logs and traces; schedule the postmortem for Monday.
Why: Closing too early risks a second wave. Evidence captured now makes the postmortem factual instead of from memory.
| Role | Does | Does not |
|---|---|---|
| Incident commander | Coordinates, sets priorities, decides on mitigation, calls for help | Debug or type commands |
| Operations lead | Investigates and changes systems, announcing every change first | Make silent changes or several at once |
| Communications lead | Updates status page, support and stakeholders on a cadence | Speculate about root cause publicly |
| Scribe | Records timestamps of observations, decisions and changes | Tidy the timeline to look better |
Interactive simulation
Visual diagram
From a symptom alert to mitigation and learning. Follow the walkthrough for a slow checkout.
Analogy
- A fire brigadeSmoke detectors where people areIncident responseAlerts on what users feel
- A fire brigadeChief on site directs, holds no hoseIncident responseThe incident commander
- A fire brigadePut out the fire, get people outIncident responseRestore service before root cause
- A fire brigadeFire investigation afterwardsIncident responseThe blameless postmortem
- A fire brigadeChanging building codes, not blaming the toaster userIncident responseFixing the system, not the person
Where the analogy breaks: firefighters rarely start the fire. In software, the outage is often caused by your own change, which is exactly why undoing that change is the fastest fix.
Check your understanding
SELECT … FROM room_availability span takes 1.8 s. Logs: Seq Scan, release v2.14.0 at 14:08. What is the root cause?14:08:02 INFO deploy checkout v2.14.0 rolled out
14:09:15 WARN db slow query 1812 ms: SELECT ... FROM room_availability WHERE hotel_id=$1 AND stay_date BETWEEN $2 AND $3 (Seq Scan)
14:10:40 ERROR checkout timeout acquiring connection from pool after 5000 msPractice ladder
Read last Tuesday's timeline
Scenario Warm-upFrom last Tuesday's postmortem timeline (checkout SLO 99.9%, ~28,000 checkouts an hour on a weekday, monthly budget 20,000 failed checkouts):
14:02 search v7.3 deployed 14:06 checkout error rate rises to 6% 14:31 burn-rate page fires (alert window was misconfigured to 6 h) 14:44 SEV2 declared; IC assigned 14:52 team spends time reading search logs 15:10 someone notices search v7.3 changed a shared library used by checkout 15:13 v7.3 rolled back 15:16 error rate back to 0.1%
Trace it: time to detect, time to mitigate, roughly how many checkouts failed, what share of the monthly budget that is, and the single biggest delay.
Finish the payment-provider runbook
Design PracticeOmar's runbook for the alert PaymentFailureFastBurn has two gaps. Fill in 3 and 5.
- Impact check: RED dashboard for payments: all providers or one? All card types or one?
- What changed? Our deploys and flags in the last 60 min. If yes, roll back or revert the flag.
- ___ (the provider's own status page shows an incident)
- Escalate: contact the provider's support with our account id and example request ids.
- ___ (who is told what, and how often)
One partner's hotels vanished from search
Scenario PracticeMonday, 09:40. Maya: "Since about 08:00, none of the 1,100 hotels from our largest channel partner show up in search." Other hotels are fine; bookings for those hotels from existing links still work. Nothing was deployed today, but the partner sync job's config was changed at 07:55 by a teammate.
You're on call. Write what you do in the first 20 minutes: severity, roles, first mitigation, and the first status update.
Should we page on database CPU?
Decision StretchIn Monday's postmortem, Karim proposes: "Page on-call whenever database CPU goes above 80%. We'd have caught Saturday earlier."
Facts: database CPU passes 80% about 15 times a week, mostly during nightly batch jobs, with no guest impact. On Saturday, the checkout burn-rate alert fired 11 minutes after the deploy; CPU crossed 80% about 4 minutes after it.
Write a short decision: "We chose X over Y because…, accepting the risk of…".
Explain it yourself
Walk through how your team should handle a slow-checkout incident from the first alert to the postmortem.