Episode 47 · Two bosses at 2 a.m.

Consensus, leader election and distributed locks

Tuesday, 02:00: the nightly payout job runs on two machines at once, each sure it holds the lock. By morning, 140 hotels have been paid twice, €61,000 in total.

How a group of machines agrees on one boss and one story, even when some crash or can't hear each other.

30 min advancedLeader electionQuorums & majoritiesRaft consensus (intuition)Distributed locks & fencing tokensCoordination services (etcd, ZooKeeper)Consistency vs availability

Problem
Exactly one scheduler instance must run the nightly invoicing job, and the database failover must promote exactly one new primary.
Why the simple system fails
'Whoever pings first wins' breaks under partitions and pauses: two nodes both think they are leader (split brain) and both write.
Engineering concept
Decide by majority. Any two majorities overlap, so two conflicting decisions can't both win. Give every leadership term a number, and let resources reject stale numbers.
Technology
Raft/Paxos-based coordination services (etcd, ZooKeeper, Consul), leases, leader election recipes, fencing tokens.
Trade-offs
Consensus costs round trips on every write, needs an odd number of nodes, and stops accepting writes when a majority is unreachable. Use it for small, critical metadata, not bulk data.

The problem

2 min

Tuesday, 07:30. Finance has opened the bank file.

Omar · On-call SRE
Payouts ran on two instances. 140 hotels paid twice, €61,000.
Karim · CTO
We put a Redis lock on that job precisely so this couldn't happen.
Omar · On-call SRE
Instance A took the lock with a 60 s timer. Its VM was paused for 90 s.

The big idea

4 min

The worker that didn't know it slept

The big idea

Decide by majority. Then make the resource refuse anyone holding an old number.

Electing one leader safely is a solved problem; you use a tool for it. But no election can wake a frozen ex-leader. So the payout table itself must check who is current, and a unique constraint makes a double payout impossible anyway.

How it works

5 min

How machines agree on one leader, and stay safe when it's stale

1
Agree by majority
: machines agree by majority vote, called a . With 3 nodes, 2 decide. Any two majorities share a node, so they can't decide two different things.
2
Leaders come in numbered terms
In , when followers stop hearing the leader's heartbeat, one bumps the term (5 → 6) and asks for votes. Each node votes once per term; a majority makes the leader. A higher term always wins.
3
Changes need a majority too
A write is final once most nodes have stored it. An old leader cut off from the majority can't finalise anything. Tools like and do all this for you.
4
A lease is a lock with an expiry
A lease is released if the holder doesn't renew it in time, whether or not the holder knows. A paused process can't feel its pause.
5
The resource checks a rising number
A rises with every new holder (A: 33, B: 34). The database remembers the highest it has seen and refuses anything older, in the same transaction as the write.

Watch a decision being made

4 min

Worked example

You fix the payouts; Lina only checks

Make it impossible to pay a hotel twice, even if two instances both believe they're the leader.

1Price a violation
Two runners means hotels paid twice: real money. A correctness lock, not an efficiency one.
Why: The cost decides the tool. If a duplicate only wasted CPU, a Redis key would do.
2Make a double run harmless first
Unique constraint on payouts(hotel_id, period), INSERT … ON CONFLICT DO NOTHING, bank transfer keyed payout-<hotel>-<period>.
Why: The best lock is the one you don't need: even two instances create each payout once.
3Let the payouts database grant the lease
A leases row updated conditionally in Postgres, where the work already lives. No new system.
Why: Coordinating in the same database as the protected writes keeps it atomic.

4Fence every write

Each payout transaction first advances the fence; zero rows means a newer holder exists.

sql

BEGIN;
UPDATE fences SET token = $1 WHERE resource = 'payouts' AND token <= $1;
-- 0 rows: ROLLBACK and stop
INSERT INTO payouts (hotel_id, period, cents) VALUES ($2, $3, $4)
  ON CONFLICT (hotel_id, period) DO NOTHING;
COMMIT;

Why: Only the resource can tell a paused ex-leader that it's stale.

The whole thing in one line: Double payouts became impossible (constraint), rare (lease) and harmless-and-visible (fence). Lina's note: alert when an instance gets fenced; it means a pause happened. Test it on staging with kill -STOP for 90 s.

In real life

3 min

A majority-backed lease versus a single-node lock

Kubernetes: leases backed by etcd

Publicly documented: cluster state lives in etcd, which uses Raft and needs a majority to write
Controllers elect one active copy with Lease objects
Won't split, but every write costs a majority round trip

Fits leadership that must never split.

Redis: a key with an expiry

Publicly documents a simple lock: SET key NX with an expiry, on one instance
Very fast, no voting
A pause or failover can let two clients act as holders

Fits efficiency locks, where a duplicate only wastes work.

What goes wrong

3 min

Treating a distributed lock as a local mutex

Pauses, clock drift and partitions make a holder lose the lock without knowing.

Instead

Fencing tokens checked by the resource, and idempotent work.

An even number of voting nodes

4 nodes need 3 votes and still survive only 1 failure, with slower writes.

Instead

3 or 5 nodes.

A quorum spread over two zones

Losing either zone can take the majority away.

Instead

Three zones, one node each: lose any zone, 2 of 3 remain.

Trade-offs

3 min

Arbiter	Fits when	You pay
Consensus service (etcd, ZooKeeper)	Leadership shared by many systems	Another critical system; majority round trips
The database itself (lease row, conditional update)	The work already lives in that database	Tied to that database's availability
Single-node lock with expiry (Redis)	Efficiency only	Can be granted twice; not for correctness

Thursday, 14:00. Where should job leadership live?

Karim · CTO
A proper 3-node etcd for all our coordination. One tool, done right.
Omar · On-call SRE
That's a new critical system on my pager at 2 a.m. Payouts already live in Postgres. Let Postgres grant the lease.
Karim · CTO
And when five more jobs need leaders, each in a different database?

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What two runners cost: double payouts is a correctness problem; two cache rebuilds only waste CPU.
What makes a double run harmless before any lock: a unique constraint and a transfer key per payout.
Sizes for anything that votes: odd numbers, spread so no single zone holds the majority.

Tell Claude Code

StayHub's payout job paid 140 hotels twice (€61,000): instance A froze 90 s, its 60 s Redis lock expired, instance B started over. Plan first. Add a unique constraint on payouts (hotel_id, period) with ON CONFLICT DO NOTHING; key each bank transfer payout-<hotel>-<period>. Replace Redis with a Postgres lease row that increments a fencing token per grant. Each payout transaction first advances a fences row only if the stored token ≤ ours; zero rows: roll back and stop. Alert when fenced. Give me a staging test with kill -STOP for 90 s.

Check before you accept

The fence check runs inside the same transaction as the payout insert.
The migration adds the unique constraint on (hotel_id, period).
In the pause test, each hotel has exactly one payout and instance A logs that it was fenced.

Red flags

It makes the lock TTL longer (60 s to 5 minutes).
It adds a stillHoldsLock() check before writing and calls the race closed.
It proposes an even-sized cluster, or two zones, for anything that votes.

Check yourself

2 min

Q1/3Scenario

A 5-node etcd cluster is split by a network partition into groups of 2 (including the old leader) and 3. What happens?

How confident are you?

Next module: StayHub's app feels slow in Jakarta even with a CDN, and it's time to look at what the network itself is doing with every packet.

Finished?

Mark it complete to update your skill map.

Next: TCP vs UDP (and why QUIC exists)

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Fencing the payout table

Worked example

You fix the payouts; Lina only checks

Make it impossible for the nightly payout job to pay a hotel twice, even if two instances both believe they're the leader.

1Price a violation
Two instances running at once means hotels paid twice: real money. This is a correctness lock, not an efficiency one.
Why: The cost decides the tool. If a duplicate only wasted CPU, a Redis key would do.
2Make a double run harmless first
Unique constraint on payouts(hotel_id, period) and INSERT … ON CONFLICT DO NOTHING. The bank transfer is sent from the payouts row with key payout-<hotel>-<period>.
Why: The best lock is the one you don't need. With this, even two instances can only create each payout once.
3Choose the arbiter
Use the payouts database itself to grant the lease (a leases row updated conditionally), since the work lives there. No new system.
Why: Coordinating in the same database as the protected writes keeps it atomic. A separate consensus cluster is worth it only if many systems need it.
4Fence every write
Each grant increments a token (33, 34…). Every payout transaction first runs UPDATE fences SET token = $1 WHERE token <= $1; zero rows → roll back and stop.
Why: A paused ex-leader can't feel its pause; only the resource can tell it that it's stale, atomically with the write.
5Prove it with a test
In staging, pause instance A for 90 s mid-run (kill -STOP, then kill -CONT) and check: no duplicate payouts, and instance A's log says "fenced, stopping".
Why: Concurrency fixes that aren't tested under the exact failure tend to have a gap nobody noticed.

sql

-- The resource enforces fencing: it remembers the highest token it accepted.
CREATE TABLE fences (resource text PRIMARY KEY, token bigint NOT NULL);

-- Every payout runs in a transaction that first advances the fence.
BEGIN;
UPDATE fences SET token = $1          -- $1 = this holder's token, e.g. 34
 WHERE resource = 'payouts' AND token <= $1;
-- 0 rows updated → a newer holder exists: ROLLBACK and stop working.
INSERT INTO payouts (hotel_id, period, amount_cents) VALUES ($2, $3, $4)
  ON CONFLICT (hotel_id, period) DO NOTHING;    -- and idempotent anyway
COMMIT;

The fence check and the write commit together, so no pause can slip between them.

Raft in five rules (intuition, not the full spec). (1) Each node is a follower, a candidate or the leader; time is split into numbered terms. (2) A follower that hears no heartbeat for a randomised timeout becomes a candidate, bumps the term and asks for votes. (3) Each node votes once per term, only for a candidate whose log is at least as up to date. (4) The leader replicates each write; it's committed once a majority has it. (5) Any node that sees a higher term steps down.

Interactive simulation

Visual diagram

Worker A

Worker B

etcd-1 (leader)

etcd-2

etcd-3

Invoices DB

Click a component

A 3-node etcd cluster, two workers competing for the 'invoicing' lock, and storage that checks fencing tokens.

Analogy

A five-person committeeA consensus cluster

A five-person committeeDecisions need three votesA consensus clusterMajority quorum
A five-person committeeTwo rooms, broken phone lineA consensus clusterA network partition
A five-person committeeOnly the room of three can decideA consensus clusterOnly the majority side can write
A five-person committee"Chair, term 12"A consensus clusterA leader with a term number
A five-person committeeBank refusing term-12 orders after term 13A consensus clusterA fencing token checked by the resource

Where the analogy breaks: committee members know when they've dozed off. A frozen machine has no idea time passed, which is exactly why the resource itself must check the token.

Check your understanding

Q1/3Reasoning

A Raft cluster has 7 nodes. How many nodes can fail while it still commits writes?

nodes

How confident are you?

Practice ladder

Step 1 · Trace

Trace a five-node split

Scenario Warm-up

A 5-node Raft cluster (n1–n5) is in term 7, with n1 as leader. A partition splits it into {n1, n2} and {n3, n4, n5}.

A client sends a write to n1. What happens to it?
n4's election timeout fires first. What does it do, and does it win?
The partition heals. n1 receives a message from the new leader. What does n1 do, and what happens to its uncommitted write?

Step 2 · Complete

Finish the fenced payout

Code Practice

Lina's fenced payout has two gaps. Fill in gap A (the fence check) and gap B (what to do when it fails), and say why the check must sit inside the transaction rather than before it.

async function payHotel(token: number, hotelId: number, period: string, cents: number) {
  await db.tx(async (t) => {
    const fenced = await t.query(
      // gap A: ___
    );
    if (fenced.rowCount === 0) {
      // gap B: ___
    }
    await t.query(
      "INSERT INTO payouts (hotel_id, period, amount_cents) VALUES ($1, $2, $3) ON CONFLICT (hotel_id, period) DO NOTHING",
      [hotelId, period, cents]);
  });
}

Step 3 · Solve

Make the inventory sync safe

Design Practice

A sync job pulls availability from partner hotels' channel manager every minute and overwrites StayHub's room_inventory table. Three copies of the job run for resilience. Today they use a Redis key sync-lock (SET NX, 30 s expiry).

Last week a copy paused for 40 s mid-sync, woke up and wrote availability it had fetched before the pause, overwriting newer numbers written by another copy. A hotel showed 4 rooms left when it had 0; two guests were sold rooms that didn't exist.

Design a safe version. Say which layer prevents what.

Step 4 · Decide

Stretch the cluster across regions?

Decision Stretch

StayHub is opening a Singapore region next to Frankfurt. Karim proposes one 3-node consensus cluster for job leadership spread across both: 2 nodes in Frankfurt, 1 in Singapore, "so we survive losing a region". The round trip Frankfurt ↔ Singapore is about 160 ms; leadership changes and lease renewals happen a few times a second.

Alternative: one 3-node cluster per region, each across that region's 3 zones, with each job owned by one region.

Write a short decision: "We chose X over Y because…, accepting the risk of…". Check Karim's claim about surviving a region loss.

Explain it yourself

Explain how a 5-node cluster elects a leader and why a network partition can't produce two leaders that both commit writes. Then explain why a lock from such a cluster still needs fencing.

0 words