Episode 60 · On for one hotel, please

Feature flags: separate deploying code from releasing features

Monday, 9:00: the new cancellation flow has been in production for a month, switched off. Maya wants it on for one hotel in Lisbon, and the only switch turns it on for all 6,000 hotels at once.

Put code on the servers today, switch it on for users whenever you like, and switch it off in seconds if it misbehaves.

30 min intermediateSeparating release from deployKill switchesPercentage rollouts and targetingExperiments behind flagsFlag debt and flag lifecycleBlast radius and cohort comparison

Problem
The booking team wants to ship a new 'pay at hotel' option. It touches checkout, takes three weeks to build, and marketing wants to launch it on a specific date in two countries first.
Why the simple system fails
A long-lived feature branch drifts and turns into a painful merge; a canary only controls which *version* runs, not which *users* see a feature; and turning the feature off means another deploy under pressure.
Engineering concept
Deploy code continuously with the new path switched off, then release it gradually by user, country or percentage with a runtime switch, and keep that switch as an instant kill button.
Technology
Feature-flag services or in-house flag tables, SDKs that evaluate rules locally, sticky hashing for percentages, trunk-based development, experiment analysis.
Trade-offs
Fine-grained, instant control in exchange for more code paths to test, a new runtime dependency, and flags that rot if nobody removes them.

The problem

2 min

Monday, 9:00.

Maya · Product manager
The new cancellation flow has been deployed for a month. Nadia's hotel in Lisbon wants to pilot it. Just her hotel.
Omar · On-call SRE
The switch is FEATURE_NEW_CANCELLATION in the environment: on means all 6,000 hotels, after a restart.
Karim · CTO
Can't we deploy a special version for her?

The big idea

4 min

Deployed is not released

The big idea

Deploying ships code; a flag decides who sees it, and can take it back in seconds.

Deploy continuously, release deliberately. The canary protects against a bad build; the flag against a bad feature. You need both.

How it works

5 min

One flag check, step by step

1
Build the context
A is a runtime switch around a code path, with rules. The service passes who the request is for: hotel htl_4471, Portugal.
2
Kill switch first
If enabled is false, the answer is off, whatever the rules say. That's the , in seconds, no deploy.
3
Rules, top to bottom
Hotel allowlist (Nadia's hotel) → on. Else Portugal at 10% → on if the hotel's bucket is below 10. Else the default: off.
4
Sticky buckets
: hash(flag name + hotel id) gives 0–99, the same every time, so a hotel never flickers between old and new.
5
Evaluated locally
The SDK holds rules in memory and answers in microseconds; changes arrive within seconds. If no rules load, the coded default (off) applies.
6
Delete it when done
: old flags left in code, each a second path to test. StayHub has 23 env-variable flags, 9 untouched for a year.

Watch a decision being made

4 min

Worked example

You plan the cancellation rollout; Lina only checks

On for Nadia's hotel this week, then Portugal, then everyone, with a way to stop in seconds and no mess left behind.

1Define the flag

Release flag, owner team-booking, expiry in 8 weeks, default off.

json

{ "key": "new-cancellation-flow", "owner": "team-booking",
  "expires": "2027-08-16", "enabled": true,
  "rules": [ { "allow": { "hotelId": ["htl_4471"] } },
             { "country": ["PT"], "percent": 10 } ],
  "default": false }

Why: An owner and an expiry stop a release flag becoming permanent debt.

2Make evaluation safe
Local evaluation from cached rules; coded default off if none load.
Why: No network call per request, and an outage can never switch an unfinished flow on.
3Make both paths safe, before the pilot
The old code learns every status the new code can write. CI tests flag on, off, and on-then-off.
Why: A flag switches behaviour, not data.
4Ramp, watch, clean up
Nadia 1 week → PT 10% → 25% → 100% → all countries, watching refunds and tickets. Two weeks at 100%: delete the flag and the old path.
Why: Each step limits who's affected; cleanup removes the second path.

In real life

3 min

Build it versus buy it

Facebook: Gatekeeper, built in-house

Has publicly described Gatekeeper, an internal system deciding who sees which features
Deeply integrated with its own data, at enormous scale
Run by dedicated infrastructure teams

Fits scale that justifies a platform of its own.

LaunchDarkly: a hosted service

A widely used commercial flag service whose SDKs evaluate rules locally
Targeting, audit logs and approvals out of the box
A subscription and a vendor to review

Fits teams wanting flags this month.

What goes wrong

3 min

Assuming a kill switch undoes everything

Rows, emails and charges from the new path remain.

Instead

Old path understands all new data before ramping; write down what the switch can't undo.

Non-sticky percentages (`Math.random() < 0.1`)

Hotels flicker between old and new.

Instead

Hash a stable id with the flag key.

Never removing release flags

Dead paths pile up; someone flips an old one years later.

Instead

Owner + expiry on every flag; CI fails on expired flags.

Trade-offs

3 min

Approach	Fits	You pay
No flags: deploy = release	Small apps, low-risk changes	Undo needs a deploy
Environment variables	Per-environment settings	Restart to change; no targeting or audit
Homemade flag table	A handful of server-side flags	Hashing, caching, audit, UI yourself
Dedicated flag service	Many teams releasing gradually	Subscription; removal discipline

Two months later. 100% everywhere; the flag is about to expire.

Maya · Product manager
Keep the flag forever. If a big chain hates the policy, I want to switch them back in seconds.
Omar · On-call SRE
Then the old cancellation code lives forever too: two paths to test on every change.
Maya · Product manager
So a chain complains and we… deploy code?

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

The flag's kind, owner and expiry: team-booking, 8 weeks.
Who sees it, in what order, and the safe default if the flag service is unreachable.
What "off" means for data already written by the new path.

Tell Claude Code

Replace the env variable FEATURE_NEW_CANCELLATION with a release flag new-cancellation-flow in our flag SDK. Owner team-booking, expiry 8 weeks. Rules in order: kill switch; allowlist Nadia's Lisbon hotel; 10% of Portuguese hotels, sticky by hotelId; else off. Evaluate locally; if the service is unavailable, off. Don't touch the other 22 variables. Check that the old code, including the refund job, handles CANCELLED_PARTIAL_REFUND with the flag off. Plan first, then tests: on, off, and off with existing partial-refund bookings.

Check before you accept

The rollout hashes hotelId, not Math.random().
A test with the flag off and existing partial-refund bookings passes, refund job included.
The fallback (off) is visible in the code.

Red flags

A network call to the flag service on every request.
"Flag off" treated as a full undo while the new path writes new statuses.
A flag used as a security control instead of real authorization.

Check yourself

2 min

Q1/3Reasoning

The new 'pay at hotel' code has been running in production for two weeks, but no guest has seen it. Which statement is accurate?

How confident are you?

Next module: a new booking service works perfectly on Omar's laptop and refuses to start on the server, and the team starts putting services in containers.

Finished?

Mark it complete to update your skill map.

Next: Docker explained

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Rules, buckets and a kill switch

Worked example

You plan the cancellation rollout; Lina only checks

Turn the new cancellation flow on for Nadia's hotel this week, then Portugal, then everyone, with a way to stop in seconds and no mess left behind.

1Define the flag
Key new-cancellation-flow, type release, owner team-booking, expiry in 8 weeks, default off.
Why: An owner and an expiry date are what stop a release flag from becoming permanent flag debt.
2Write the rules
1) kill switch (enabled); 2) allow hotel htl_4471; 3) country PT at 10%, bucketed by hotelId; 4) default off.
Why: Order matters: the kill switch beats everything, the pilot hotel is guaranteed in, and buckets by hotel keep each hotel consistent.
3Make evaluation safe
SDK evaluates locally from cached rules; if no rules can be loaded at start-up, the coded default (off) applies.
Why: No network call per request, and a flag-service outage can never switch an unfinished flow on.
4Make both paths safe
Before the first ramp, the old code learns every status the new code can write. CI runs the cancellation tests with the flag on, off, and on-then-off.
Why: A flag switches behaviour, not data. Lina's only check: this step must happen before Nadia's pilot, not after.
5Ramp, watch, clean up
Nadia 1 week → PT 10% → 25% → 100% → all countries, each step 3–5 days, watching refund amounts and cancellation-related tickets. Two weeks at 100%: delete the flag and the old path.
Why: Each step limits who's affected; the cleanup removes the second code path once it's no longer needed.

import { createHash } from "node:crypto";

type Rule =
  | { kind: "allow"; hotelIds: string[] }
  | { kind: "country"; values: string[]; percent: number };

interface Flag { key: string; enabled: boolean; rules: Rule[]; defaultOn: boolean }
interface Ctx { hotelId: string; country: string }

function bucket(flagKey: string, id: string): number {
  const h = createHash("sha256").update(`${flagKey}:${id}`).digest();
  return h.readUInt32BE(0) % 100; // 0..99, stable for this hotel and flag
}

export function isEnabled(flag: Flag | undefined, ctx: Ctx, fallback = false): boolean {
  if (!flag) return fallback;              // unknown flag or no rules loaded: safe default
  if (!flag.enabled) return false;         // kill switch wins over every rule
  for (const r of flag.rules) {
    if (r.kind === "allow" && r.hotelIds.includes(ctx.hotelId)) return true;
    if (r.kind === "country" && r.values.includes(ctx.country)) return bucket(flag.key, ctx.hotelId) < r.percent;
  }
  return flag.defaultOn;
}

A minimal, deterministic evaluator. Real SDKs add streaming updates, segments and exposure events.

Interactive simulation

Visual diagram

Flag dashboard

Flag service

Guest

Booking API + flag SDK

Pay-at-hotel path

Classic checkout

Metrics by variant

Click a component

How a flag decision is made on every request without a network call. Click components or start the walkthrough.

Analogy

Wiring a new hotel wingFeature flags

Wiring a new hotel wingWires in the walls, breaker offFeature flagsCode deployed, feature not released
Wiring a new hotel wingTurn on one floor firstFeature flagsGradual rollout to some users
Wiring a new hotel wingA circuit sparks: flip its breaker offFeature flagsKill switch, no redeploy
Wiring a new hotel wingOld test switches left in the panelFeature flagsFlag debt: forgotten flags pile up

Where the analogy breaks: flipping a breaker restores the lights instantly. Turning off a flag doesn't undo data the feature already wrote, like bookings made with the new option.

Check your understanding

Q1/3Design

You add a kill switch disable-hotel-recommendations that turns off calls to a flaky recommendation service. The flag service itself becomes unreachable during an incident. What should the SDK do?

How confident are you?

Practice ladder

Step 1 · Trace

Evaluate the flag for four hotels

Scenario Warm-up

Using Lina's isEnabled and this flag:

{ "key": "new-cancellation-flow", "enabled": true,
  "rules": [ { "kind": "allow", "hotelIds": ["htl_4471"] },
             { "kind": "country", "values": ["PT"], "percent": 25 } ],
  "defaultOn": false }

Bucket values for this flag: htl_4471 → 63, htl_2001 → 12, htl_3050 → 88, htl_9100 → 7.

Hotel	Country
htl_4471	PT
htl_2001	PT
htl_3050	PT
htl_9100	ES

Which hotels get the new flow?
Same question after someone sets "enabled": false.
Same question if the service starts and can't load any rules (flag undefined, fallback false).

Step 2 · Complete

Make the old path understand the new data

Code Practice

The refund job's old code crashed on the new status. Fill in the gap so the old path handles every status the new flow can write, and say in which release this must ship relative to turning the flag on.

type Status = "CONFIRMED" | "CANCELLED" | "CANCELLED_PARTIAL_REFUND";

function refundAmount(b: { status: Status; totalCents: number; refundCents?: number }): number {
  switch (b.status) {
    case "CONFIRMED":
      return 0;
    case "CANCELLED":
      return b.totalCents;
    // gap: ___
    default:
      throw new Error(`unknown status ${b.status}`);
  }
}

Step 3 · Solve

Launch "pay at hotel"

Design Practice

You own the launch of "pay at hotel" (guests reserve now, pay at check-in). Requirements:

Code merged daily for three weeks before launch.
Launch on 1 March in the Netherlands and Belgium only, then expand.
A guest must see a consistent checkout across visits and devices.
Checkout revenue is critical: if no-shows rise, it must be off within a minute.
The feature adds a payment_mode column to bookings.

Design the flag (key, type, owner, expiry, rules, bucketing id, default), the data steps, and the launch and cleanup plan.

Step 4 · Decide

Fail safe or fail open?

Decision Stretch

During an incident, the flag service may be unreachable when a service starts, so its SDK has no rules and falls back to coded defaults. Two flags need a decided fallback:

disable-recommendations: a kill switch that, when on, stops calls to the recommendations service (which sometimes gets slow and once took hotel pages down).
new-cancellation-flow: now at 100% everywhere, about to be removed.

For each, choose the fallback (on or off) and write one decision: "We chose X over Y because…, accepting the risk of…".

Explain it yourself

Explain the difference between a canary deploy and a feature-flag rollout, and when you would use each, or both.

0 words