Skip to content
Episode 89 · 38 minutes in public

Secrets, hashing and encryption

Wednesday, 11:02: a scanner alert says StayHub's live payment-provider key has been sitting in a contractor's public code repository for 38 minutes. The contractor has already deleted it. Is StayHub safe?

Encoding, hashing and encryption sound alike but do very different jobs. Plus: where your app's passwords and keys should live.

30 min intermediateSecrets managementSecret rotation and leak responseHashing vs encryption vs encodingPassword hashing with slow salted functions (Argon2, bcrypt)Encryption in transit vs at restKMS and envelope encryption
  1. Problem

    StayHub holds a payment provider API key, database passwords, guests' passport numbers and millions of password hashes. Any of them leaking is a headline.

  2. Why the simple system fails

    Secrets hardcoded in the repo end up in every clone, CI log and laptop. Unencrypted backups leak whole databases. Fast hashes like SHA-256 let a stolen password table be cracked in hours. Keys that never rotate stay valid long after someone leaves.

  3. Engineering concept

    Keep secrets out of code and fetch them at runtime from a controlled store; rotate them routinely and on leaks. Hash what you only need to verify (passwords) with a slow salted function; encrypt what you need to read back, with keys managed separately from the data.

  4. Technology

    Secrets managers, CI secret stores and secret scanning, TLS for transit, disk/database/backup encryption at rest, KMS with envelope encryption for sensitive fields, Argon2id or bcrypt for passwords.

  5. Trade-offs

    Every layer adds latency, cost and operational risk (lose the key, lose the data). Field-level encryption protects against more threats but breaks search and indexing. Rotation is safest when automated, and automation needs dual-credential support in the app.

01

The problem

2 min
Wednesday, 11:02. A secret-scanning alert lands in the security channel.
  1. Omar · On-call SRE
    Our live payment-provider secret key is in a contractor's public demo repository. Pushed at 10:24.
  2. Omar · On-call SRE
    11:10: the contractor deleted the file and force-pushed. History is clean.
  3. Karim · CTO
    So we're fine?
02

The big idea

4 min
Deleted is not safe
The big idea

A leaked secret is compromised forever, so rotate first; and for stored data, one question picks the tool: do we ever need the original back?

Think of the hotel's key cabinet at reception: locked, logged, signed out per shift. A key taped under the desk is a secret in the code. Re-keying the locks when a key goes missing is rotation.

03

How it works

5 min
Secrets, then stored data
  1. 1
    Secrets live in a secret manager

    A is a locked, logged store that services read at runtime. Never in code, a committed .env or a Docker image.

  2. 2
    Rotate without an outage

    : 11:14 new key at the provider (the old one still works); secret manager version 18; services reload; 11:21 revoke the old key. New before revoking, so payments never fail.

  3. 3
    Read the logs, close the path

    Provider logs: 3 calls from an unknown address (balance, last 100 charges). No card numbers: the provider tokenizes them. Then secret scanning on every commit.

  4. 4
    Only compare? Hash

    is one-way: you can check a match, never get the input back. Passwords need a slow, salted hash like ; tokens we issue can use fast SHA-256 because they're long and random.

  5. 5
    Need it back? Encrypt

    Encryption is two-way with a key. Disk decrypts for anyone who can query the database, so SQL injection still sees everything.

  6. 6
    Most sensitive fields: envelope encryption

    : each record gets its own data key; a holds the master key that wraps them and logs every use. The database alone can't read the field.

04

Watch a decision being made

4 min
Worked example

You run the secrets audit, Lina checks

After the leak, protect everything sensitive StayHub holds, so a stolen database dump, log or key does as little harm as possible.

  1. 1Ask one question per value

    Need the original back? Passwords: no. Passports: yes, at check-in. Calendar refresh tokens: yes. Partner API tokens we issue: no. Card numbers: we never had them.

    Why: Lina: "That one question picks the tool. Most mistakes come from skipping it."

  2. 2Hash what we only compare

    Passwords: keep verifying bcrypt, rehash with Argon2id on each successful login. Partner tokens: store SHA-256, show the token once.

    Why: A database dump then contains no working tokens and no fast-crackable passwords.

  3. 3Envelope-encrypt passports

    Encrypt the 410,000 plaintext passport numbers, each with a fresh data key; store the KMS-wrapped key beside the ciphertext. Only the check-in service may unwrap.

    Why: A field the database can't read survives SQL injection and over-curious support tools.

  4. 4Narrow and rotate the credentials

    Restricted provider keys per service (refunds can't create charges), all in the secret manager. Database passwords rotate every 30 days.

    Why: The next leak should expose one narrow key, and a rotation done monthly still works in an emergency.

The whole thing in one line: You sorted every value by "do we need it back?", hashed what we only compare, envelope-encrypted what we must read, left cards to the specialist, and made credentials narrow and routinely rotated.
05

In real life

3 min
Static keys vs short-lived identity

Cloud access keys: long-lived, rotated

  • AWS documents long-lived access keys, with guidance to rotate them and prefer temporary credentials
  • If copied, it works for the attacker until someone rotates it

Easy to start; every copy is a standing risk.

GitHub Actions: OIDC federation

  • Publicly documents OIDC so workflows get short-lived cloud credentials per run
  • No static cloud key stored in CI; trust rules set on the cloud side

Nothing long-lived to leak; a little more setup.

06

What goes wrong

3 min
Deleting a committed secret and moving on

It's in forks, caches and bots' collections already.

Instead

Rotate first, then clean up, then check usage logs.

Hashing passwords with SHA-256, even salted

Attackers test billions of guesses a second.

Instead

Argon2id (or bcrypt), rehashing on login.

Storing the encryption key next to the data

One leak of the database or config exposes both.

Instead

Master keys in a KMS with separate permissions and audit logs.

07

Trade-offs

3 min
OptionProtects againstYou pay
Disk encryption at restStolen drives, some snapshot leaksLittle; always on. No help against SQL injection
Field-level envelope encryptionDumps, SQL injection, curious toolsNo search or sort on the field; KMS latency and cost; lose the key, lose the data
Secret managerSecrets copied around in code and imagesA startup dependency; cost per secret or call
Friday, 14:00. Reviewing your audit.
  1. Karim · CTO
    The database is already encrypted at rest. Field encryption means KMS calls, no searching, and if we lose the key we lose the data.
  2. Lina · Senior engineer, your mentor
    At-rest encryption won't stop a SQL injection. The database hands the plain value to anyone who can query it.
  3. Karim · CTO
    Support searches by passport number when a guest calls.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • What each kind of data gets: passwords , passports , cards tokenized by the provider. An agent tends to pick one tool for everything.
  • Where secrets live: the , fetched at runtime. Never in code, a committed , an image or a chat prompt.
  • The order when a key leaks: rotate first, then check the logs, then clean history.
Tell Claude Code
We store 410,000 passport numbers in a plaintext column. Plan first, then wait. Propose field-level envelope encryption: per passport, a KMS data key, AES-GCM in the app, ciphertext and wrapped key in new columns. Write a resumable batch backfill that verifies each row decrypts before the old column is cleared. Read the KMS key id from configuration. Don't touch password or payment code. Test that stored values aren't passport numbers, that decryption round-trips, and that no passport reaches a log.
Check before you accept
  • On staging, the new column is unreadable, and the plaintext column is emptied only after verification.
  • The master key lives in the KMS; only wrapped data keys are stored.
  • The encryption is a standard AES-GCM call with a fresh nonce per value, not a homemade scheme.
Red flags
  • It "encrypts" passports with Base64. That's encoding; anyone can reverse it.
  • It hashes passport numbers. You need them back at check-in.
  • It hard-codes a key in the source or a committed .env "for now".
09

Check yourself

2 min
Q1/3Decision
Match each piece of StayHub data to the right protection:
Guest login password
Passport number for check-in
Credit card number
Webhook payload from a partner
How confident are you?

Next episode: an attacker gets into one forgotten internal tool, and from there everything inside the network trusts them.

Finished?

Mark it complete to update your skill map.

Next: Zero trust: identity over network location
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Hash it, encrypt it, or don't keep it
Worked example

You run the secrets audit, Lina checks

After the leak, list everything sensitive StayHub stores or holds, and protect each one so that a stolen database dump, log or key does as little harm as possible.

  1. 1Ask one question per value

    Do we need the original back? Passwords: no. Passports: yes. Refresh tokens: yes. Partner API tokens we issue: no. Card numbers: we never had them.

    Why: Lina: "That one question picks the tool. Most mistakes come from skipping it."

  2. 2Hash passwords slowly

    Keep verifying bcrypt hashes; on each successful login, rehash with Argon2id.

    ts
    if (!(await verifyPassword(user.passwordHash, password))) throw new InvalidCredentials();
if (user.passwordHash.startsWith("$2b$")) {       // old bcrypt hash
  await saveHash(user.id, await argon2.hash(password, { type: argon2.argon2id }));
}

    Why: Fast hashes like SHA-256 let a GPU test billions of guesses a second. Slow, salted, memory-hard hashing makes a stolen table far less useful.

  3. 3Envelope-encrypt passports

    Encrypt the 410,000 passport numbers with a fresh data key each; store the KMS-wrapped data key next to the ciphertext. Only the check-in service may unwrap.

    Why: Disk encryption decrypts for anyone who can query the database. A field the database can't read survives SQL injection and over-curious support tools.

  4. 4Hash the tokens we issue

    Partner API tokens: store SHA-256 of the token, compare hashes on each request. Show the token to the partner once.

    Why: We only need to compare. A long random token can't be guessed, so a fast hash is fine, and a database dump no longer contains working tokens.

  5. 5Scope and store the credentials

    Restricted provider keys per service (refunds can't create charges), all in the secret manager; CI moves to OIDC federation.

    Why: The next leak should expose one narrow key, and CI should hold nothing long-lived.

  6. 6Make rotation routine

    Database passwords rotate every 30 days with two users: set a new password on the inactive user, publish it as the next version, let connections drain, switch next time.

    text
    rotation N:   app_b gets new password -> secret v19 = app_b -> apps reconnect -> app_a idle
rotation N+1: app_a gets new password -> secret v20 = app_a -> apps reconnect -> app_b idle

    Why: A rotation done every month works in an emergency. One done once a year breaks things the day you need it.

The whole thing in one line: You sorted every sensitive value by "do we need it back?", hashed what we only compare, envelope-encrypted what we must read, kept what a specialist can hold out of our systems, and made credentials narrow, stored centrally and rotated routinely.
11

Interactive simulation

12

Visual diagram

Click a component

Secrets flow at runtime; sensitive fields use envelope encryption; passwords are only ever hashed.

13

Analogy

  • The hotel's keys and safesLocked, logged key cabinet at receptionYour appA secrets manager
  • The hotel's keys and safesKey taped under the deskYour appA password hardcoded in the code
  • The hotel's keys and safesRe-keying locks when staff leaveYour appRotating secrets
  • The hotel's keys and safesStoring a fingerprint of your safe codeYour appHashing passwords
  • The hotel's keys and safesSmall locked boxes, master safe for their keysYour appEnvelope encryption with a KMS

Where the analogy breaks: a stolen hotel key is missing from the cabinet, so you notice. A copied secret looks exactly like the original, so you may never know. That's why rotation matters.

14

Check your understanding

Q1/4Scenario
A developer pushed the production payment API key to a public repository 20 minutes ago. Order the response:
  1. 1.Remove the key from the repository history
  2. 2.Add pre-commit and CI secret scanning; move remaining config secrets out of the repo
  3. 3.Revoke/rotate the key at the payment provider and deploy the new one from the secrets manager
  4. 4.Check the provider's logs for calls made with the old key since the push
How confident are you?
15

Practice ladder

Step 1 · Trace

Sort StayHub's sensitive values

Decision Warm-up

For each value, trace the question "do we need the original back?" and pick the protection: slow hash, fast hash, encrypt (envelope/KMS), don't store (tokenize), or HMAC signature.

  1. A guest's login password
  2. A hotel owner's bank account number (for payouts)
  3. The card number a guest pays with
  4. A password-reset token we email to a guest (64 random characters)
  5. A webhook the payment provider sends us about a refund
Step 2 · Complete

Finish the database password rotation

Design Practice

The booking service connects to PostgreSQL. Lina's zero-downtime rotation plan uses two database users, app_a (currently in use) and app_b (idle). Fill in steps 2 and 4.

  1. Generate a new password and set it on app_b in the database.
  2. ___
  3. Booking servers fetch the new secret version on their next refresh (every 5 minutes) and open new connections as app_b.
  4. ___
  5. Next month, repeat with the roles swapped.
Step 3 · Solve

Encrypt the hotel bank accounts

Design Practice

StayHub pays 9,000 hotels every week and stores each hotel's bank account number (IBAN) in plain text. The payouts job reads all of them once a week; the extranet shows the last 4 characters to hotel staff; support sometimes needs to confirm an account when a hotel calls.

Design the protection: what's encrypted and how, who may decrypt, what support sees, and what's logged.

Step 4 · Decide

How long should we keep passports?

Decision Stretch

Passports are now envelope-encrypted. Maya wants to keep them 2 years so returning guests skip the passport step (about 30% of guests come back within 2 years). Omar wants to delete them 30 days after checkout, the legal minimum for some hotels' guest registers.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain how StayHub should handle a guest's password, passport number and the payment API key, and why each gets a different treatment.

0 words