Episode 56 · The hotfix that came undone

Git, branching and code review

Monday, 9:00: two engineers spend the whole day merging a three-week-old branch. By evening, a VAT hotfix everyone saw "in main" is missing from production again, and 1,240 invoices are off by a cent.

How a team of people can change the same code every day without stepping on each other, and why small changes win.

30 min foundationGit commits & branchesMerge vs rebaseTrunk-based development vs GitFlowPull requests, code review & small batches

Problem
Eight engineers change the same booking codebase every day without overwriting each other, and production must always be shippable.
Why the simple system fails
Sharing code by copying folders or editing one server loses work. Long-lived personal branches drift for weeks, then explode into giant merge conflicts and unreviewable changes.
Engineering concept
Version control records every change as commits; branches let work proceed in parallel; integrating small changes often keeps everyone close to the shared main line.
Technology
Git, pull requests on a Git host, merge and rebase, trunk-based development, GitFlow, feature flags.
Trade-offs
Strict, long-lived branch models give release control but slow integration; trunk-based flow is fast but demands good tests, small changes and feature flags.

The problem

2 min

Monday, 17:00. Two engineers have been merging since 9:00.

Omar · On-call SRE
The search filters branch: 3 weeks old, 4,100 lines, 63 conflicted files.
Maya · Product manager
And 1,240 invoices are off by a cent again. The VAT hotfix went to main two weeks ago.
You · Engineer at StayHub
How can a fix that's in main disappear?

The big idea

4 min

How a fix came undone

The big idea

Branches are free; divergence is what costs. Merge small changes often.

"Be more careful with conflicts" won't work: 63 files at 6 p.m. always produce a mistake. The fix is to have far fewer conflicts, by spending less time apart.

How it works

5 min

From snapshots to one trunk

1
Commit
A is a snapshot of the whole project, with an author, a message and a pointer to the commit before it.
2
Branch
A is a movable bookmark pointing at a commit. Creating one is free. About 30 commits land each day, so a 3-week branch drifts ~450 commits away.
3
Merge or rebase
A joins two histories with a joining commit. A rebase replays your commits on top of the latest main, with new ids. Rebase only your own branch; never rewrite commits others already have.
4
Pull request
A asks to merge a branch into main: tests run and a teammate reviews.
5
Trunk-based development
: one always-releasable main, branches of a day or two, small PRs reviewed within hours, unfinished work merged switched off behind flags. A hotfix is just another PR into main.
6
Where GitFlow still fits
(develop, release and hotfix branches) fits products with several versions in use, like our mobile app.

Watch a decision being made

4 min

Worked example

Lina moves the web team to trunk-based development

Stop multi-day merges and lost fixes without slowing anyone, while the search filters (3 more weeks) keep moving.

1Measure the pain
Median branch age 9 days, median PR 2,300 lines, review wait 2 days, 2 incidents this quarter.
Why: Numbers make the change concrete and checkable in a month.
2Set the rules
Branches ≤ 2 days. PRs under ~400 lines, one purpose each. Review within 4 working hours.
Why: A small PR that waits two days becomes a two-day-old branch. The whole loop must be short.

3Protect main, split the big feature

No direct pushes; green CI and one approval to merge. Search filters become small PRs, each behind the search-filters flag, off in production.

bash

git switch main && git pull
git switch -c search/price-filter-api
git commit -m "Add min/max price to search API"
git fetch origin && git rebase origin/main
git push --force-with-lease   # my branch only
# open PR: ~180 lines, CI green, 1 review

Why: Merged isn't released: the feature moves daily, invisible to guests.

4Fix the hotfix path
Every fix is a PR into main. Mobile release branches get cherry-picks only from main.
Why: With one line of history, a fix can't live where it's never merged back.

In real life

3 min

One trunk versus many trees

Google: one monorepo, trunk-based

Has publicly described most code in one repository, committed to the head of the trunk
Small changes, heavy automated testing, review before each commit
Unfinished work hidden behind flags

Fits one organisation shipping continuously.

Linux kernel: maintainers' trees

Publicly develops through subsystem trees, merged during each release's merge window
Several versions maintained for years
Release branches are essential

Fits versioned releases from many independent groups.

What goes wrong

3 min

Long-lived feature branches

Divergence grows daily; merges become conflict sessions and fixes get lost.

Instead

Merge to main every day or two, behind flags.

Rebasing or force-pushing a shared branch

Teammates' copies point at commits that no longer exist.

Instead

Rebase only your own branch; --force-with-lease; merge shared ones.

Giant PRs mixing refactor, formatting and features

Reviewers can't find the risky lines.

Instead

One purpose per PR.

Trade-offs

3 min

Approach	Fits	You pay
Trunk-based development	Web and SaaS, one production version	Fast CI, good tests, flags and their clean-up
GitFlow / release branches	Mobile apps, libraries, on-premises	Long branches; fixes lost between lines
Straight to main	Solo or 2-person prototypes	No review, no gate

Tuesday, 10:00. Six weeks before summer peak.

Karim · CTO
Not the time to change how 20 engineers work. Keep GitFlow, add a release manager who checks every merge.
Lina · Senior engineer, your mentor
A release manager checking 63 conflicted files is still a person at 6 p.m. making 63 decisions.
Karim · CTO
And if trunk-based breaks main during peak?

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

How big one change may be: under ~400 lines, under 2 days. An agent will happily write 4,000 lines in an afternoon.
Where unfinished work waits: merged to main behind a flag, not on a drifting branch.
Which Git actions need your OK: force-push, history rewrites, merging to main, deleting branches.

Tell Claude Code

Create feature/search-price-filter from the latest main and add the price filter to hotel search, behind the flag search_price_filter (off by default). One PR under about 400 lines: if it won't fit, stop and propose a split into PRs each safe to merge alone. Don't reformat or rename unrelated code. Small commits, clear messages. Never force-push or commit to main. Check the diff for keys, tokens or .env files. Run tests and lint, and write a PR description: what, why, how tested, how to turn the flag on.

Check before you accept

git diff --stat: under ~400 lines, only files that belong to the feature.
Claude scanned the diff and the commit history for secrets, not just the latest file.
CI checks are green, not skipped.

Red flags

git push --force on a shared branch.
It "removes" a committed key in a new commit and calls it fixed.
It resolves a conflict by taking one side wholesale without saying what the other side contained.

Check yourself

2 min

Q1/3Debugging

Two developers share feature/checkout. Alice rebases it onto main and force-pushes. Bob then pulls and sees duplicate commits and conflicts on work he didn't touch. Why?

alice$ git rebase origin/main
alice$ git push --force
bob$   git pull
# CONFLICT ... duplicate commits 'Add card form', 'Validate CVV'

How confident are you?

Next episode: a change passes every test on Lina's laptop, gets merged, and checkout breaks in production twenty minutes later.

Finished?

Mark it complete to update your skill map.

Next: CI/CD pipelines

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

One trunk, small changes

Worked example

Lina moves the web team to trunk-based development

Stop multi-day merges and lost fixes in the web repo without slowing anyone down, while the search filters (3 more weeks of work) keep moving.

1Measure the pain
From the Git host: median branch age 9 days, median PR 2,300 lines, median wait for review 2 days, 2 incidents this quarter from lost or mis-merged changes.
Why: Numbers make the change concrete, and give you something to check in a month.
2Set the rules
Branches ≤ 2 days. PRs under ~400 lines, one purpose each (refactors separate from behaviour changes). Review within 4 working hours.
Why: Small, fast PRs are what make short branches possible. A rule nobody can follow because reviews take days just moves the queue.
3Protect main
Branch protection: no direct pushes; merging needs green CI (lint, types, tests) and one approval. Squash-merge, delete the branch.
Why: If main is always releasable, every merge must pass the same gate. Automation catches the nitpicks so reviewers discuss design.
4Split the big feature
Search filters become a stack of small PRs: API fields, query changes, UI components, each merged behind a flag search-filters that's off in production.
Why: Merged isn't released. The feature keeps moving daily without anyone seeing half of it.
5Fix the hotfix path
Every fix, urgent or not, is a PR into main. Only the mobile app's release branches get cherry-picks, and only after the fix is in main.
Why: One line of history means a fix can't live on a branch that's never merged back.

	Merge	Rebase
What it does	Creates a merge commit with two parents	Rewrites your commits on top of the target, with new ids
History	True record; can look tangled	Linear and easy to read
On shared branches	Safe: never rewrites commits	Dangerous: others' copies point at commits that no longer exist
Typical use	Merging PRs; syncing shared branches	Updating your own branch before review

Golden rule: never rebase commits that others have already pulled. Use --force-with-lease when you push a rebased branch.

Visual diagram

Developer

Local repo

Git host (PR)

CI runner

Reviewer

main branch

Production

Click a component

The life of one change in a trunk-based team using pull requests.

Analogy

Writing a guidebook togetherWorking with Git

Writing a guidebook togetherA saved draft with a noteWorking with GitA commit
Writing a guidebook togetherA bookmark: "my draft continues here"Working with GitA branch
Writing a guidebook togetherFolding your chapter into the master copyWorking with GitA merge
Writing a guidebook togetherHanding your chapter to an editor firstWorking with GitA pull request with review
Writing a guidebook togetherAn author vanishing with 60 pagesWorking with GitA long-lived branch

Where the analogy breaks: copying a book takes effort. In Git, a branch costs nothing, so there's no excuse not to make one for every small change.

Check your understanding

Q1/4Reasoning

Creating a branch in Git copies all the project's files, so you should avoid creating too many branches.

How confident are you?

Practice ladder

Step 1 · Trace

Trace a rebase on a shared branch

Scenario Warm-up

main has commits A → B → C. Alice and Bob share the branch search/filters, which has D → E on top of B (not C). Both have pulled D and E.

Alice runs git rebase origin/main and git push --force. Draw the branch on the server afterwards.
Bob, who has a new local commit F on top of E, runs git pull. What does he see, and why?
What should Alice have done instead?

Step 2 · Complete

Finish the team's PR checklist

Design Practice

Lina's checklist for every pull request into main has two blank items. Fill them in and give the reason for each.

Branch created from the latest main, less than 2 days old.
One purpose only; refactors and formatting in a separate PR.
___
Description says why, what changed, how it was tested, and what's risky.
___
Squash-merge, then delete the branch.

Step 3 · Solve

The partner API team's merge hell

Design Practice

The 6-engineer partner API team still works the old way:

Feature branches live about 3 weeks.
PRs average 2,500 lines and wait 3 days for review.
Every second Thursday, a release merge takes the whole afternoon.
Twice this quarter, a fix made on a release branch was missing from the next release.
CI takes 25 minutes.

Plan their move to trunk-based development over the next month. What do you change first, and why in that order?

Step 4 · Decide

Two approvals on every PR?

Decision Stretch

After the lost hotfix, Karim proposes: every PR into main needs two approvals instead of one.

Facts: ~60 PRs a day across the web repo; median review wait is now 3 hours with one approval; the lost hotfix was caused by a conflicted release merge, not by a badly reviewed PR. Payment and authentication code is touched by about 5 PRs a day.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain to a new teammate why your team keeps branches short and PRs small, and how you ship features that take three weeks to build.

0 words