Episode 21 · Checkout takes six seconds

Background jobs and queues

Monday, 9:30: checkout takes 5.8 s because the booking request also sends the email, renders the invoice and syncs the channel manager. On Saturday the email provider was down for 20 minutes, and 41 guests who had already paid saw an error.

How to answer the user in a blink while the slow stuff, like emails and PDFs, gets done a moment later.

30 min intermediateBackground jobs, queues and workersJob retries, backoff and dead-letter queuesCron and scheduled jobsIdempotencyAt-least-once deliveryQueueing: why latency explodes near saturation

Problem
A booking request must also send a confirmation email, generate a PDF invoice and notify the hotel's channel manager. Users wait 6 seconds and sometimes see errors.
Why the simple system fails
Doing everything inside the request ties the user's latency and success to the slowest, flakiest dependency. One email-provider outage fails bookings that already succeeded.
Engineering concept
Record the intent as a job, answer the user immediately, and let workers do the work later with retries, backoff and a place for jobs that keep failing.
Technology
A queue (Redis-based like BullMQ/Sidekiq, a managed queue like SQS, or a Postgres jobs table), worker processes, a scheduler for cron jobs, a dead-letter queue.
Trade-offs
Faster, more resilient requests in exchange for eventual completion, at-least-once execution (duplicates), more moving parts, and the need to monitor a backlog.

The problem

2 min

Monday, 9:30. Maya has the conversion chart open.

Maya · Product manager
Checkout takes 5.8 s. Conversion dropped from 3.1% to 2.6%.
Omar · On-call SRE
And Saturday the email provider was down 20 minutes. 41 guests paid and then got a 500.
Lina · Senior engineer, your mentor
You draft the plan; I'll check it. First, a guess.

The big idea

4 min

Decide now, do it later

The big idea

The request decides and records. Workers do the work later.

Separate what must happen now from what must happen soon. Then an email outage becomes a delay, not an error.

How it works

5 min

Before "Confirmed": authorize the payment and insert the booking, 400 ms. Soon: the email (300 ms), the invoice PDF (1,500 ms), the channel manager sync (~3.6 s, often timing out). A is a small note describing work to do later, like { type: "send_email", bookingId: 981 }.

What happens with jobs

1
Do the essential part
Charge the card, insert the booking: ~400 ms.
2
Record the rest
Insert jobs into a queue, where they wait in order until taken. Each carries only the booking id.
3
Answer
Return 201 Confirmed. The guest is done.
4
Workers pull
A , a separate process, takes the next job, loads the booking and does the work.
5
Retry or park
Failed? Wait longer each time and retry. After 8 failures, the job moves to the for a human, with an alert.

Checkout drops from 5.8 s to ~0.4 s; the email arrives ~3 s later. Some jobs run on a timer: "release unpaid holds every 5 minutes" is a .

Watch a decision being made

4 min

Worked example

You draft the job plan, Lina checks it

Move the side work out of checkout without losing any of it, using the PostgreSQL StayHub already runs.

1Record jobs with the booking

Insert the jobs in the same transaction as the booking.

await db.transaction(async (tx) => {
  const b = await tx.bookings.insert({ roomId, guestId, status: "confirmed" });
  await tx.jobs.insert({ type: "send_confirmation_email", payload: { bookingId: b.id } });
  await tx.jobs.insert({ type: "render_invoice", payload: { bookingId: b.id } });
});

Why: Lina's check: enqueueing after commit can fail silently (a booking with no email). Here both exist or neither.

2Let workers claim jobs safely
Two workers claim jobs with FOR UPDATE SKIP LOCKED and a 60-second lock. If a worker dies, the lock expires and the job is available again.
Why: Workers take different jobs without waiting on each other, and no job is lost to a crash.
3Shape the retries
Timeouts and 503s retry with (1 s, 2 s, 4 s… capped at 5 minutes). An invalid address doesn't retry. After 8 attempts: dead-letter table and alert.
Why: Instant retries hammer a provider that's down; a limit stops a looping forever.
4Safe to run twice; watch the oldest job
Before sending, insert a unique confirm:<bookingId> row; if it exists, stop. Alert when the oldest queued job is over 2 minutes old.
Why: Lina's second check: a crash after sending means it will run again. "Emails 25 minutes late" is what guests feel.

In real life

3 min

Two well-known teams, two places for jobs

GitHub: jobs in Redis (Resque)

Publicly described building Resque, a Redis-backed queue, around 2009
Fast, many jobs per second
A separate system; durability depends on its settings

Fits high volume and a team happy to run Redis.

37signals: jobs in the database (Solid Queue)

Publicly released Solid Queue, keeping jobs in the app's SQL database
No extra system; a job can share the booking's transaction

Fits fewer moving parts at moderate volume.

What goes wrong

3 min

Assuming each job runs exactly once

Crashes and lock timeouts cause redelivery: double emails, double refunds.

Instead

Idempotent jobs: unique keys, provider .

One queue for everything

A newsletter burst delays the confirmations guests wait for.

Instead

Separate queues or priorities by business impact.

Cron on every app server

With 3 servers, the nightly invoice job runs 3 times.

Instead

One scheduler or a lock, and idempotent jobs anyway.

Trade-offs

3 min

Option	Fits when	You pay	Main risk
In the request	Fast, reliable, result needed now (payment)	Latency of every step	One failure fails all
Background job	Emails, PDFs, partner syncs	A UI that says "later"	Duplicates, silent backlogs
Event log (Kafka)	Many teams react to the same events	Heavy operations	Overkill for a few job types

Wednesday, 10:00. Checkout is down to 0.4 s, and Maya wants more.

Maya · Product manager
Payment is still 300 ms of the 400. Put it in a job too: "Confirmed" instantly, charge later.
Omar · On-call SRE
Then we confirm unpaid bookings. Cards get declined after the guest has left.
Maya · Product manager
We'd email them. Most cards go through.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

What the guest waits for (payment and the booking row, ~0.4 s) and what becomes a job (email, invoice PDF, channel-manager sync).
How late a job may be before someone is alerted: the email normally takes ~3 s, so a few minutes is already a problem.
The retry policy (8 attempts, backoff with jitter, then the dead-letter queue), and who looks at that queue and when.

Tell Claude Code

Checkout takes ~5.8 s doing everything inline: payment and insert (400 ms), email (300 ms), invoice PDF (1.5 s), channel-manager sync (~3.6 s). When the email provider was down 20 minutes, 41 paid bookings showed an error. Keep payment and the insert in the request; move the rest to jobs in our Postgres jobs table, 2 workers, max 8 attempts with backoff and jitter, then a dead-letter queue. Write jobs in the booking's transaction. Make the email job safe to run twice. Plan first, then test a worker killed after sending.

Check before you accept

Checkout takes about 0.4 s, and the email still arrives within seconds.
A worker killed after sending but before marking the job done: the guest gets exactly one email.
A provider failing every time: the job reaches the dead-letter queue after 8 attempts, with an alert.

Red flags

Jobs enqueued after committing the booking, in a separate step.
No duplicate protection, assuming exactly-once delivery.
Retries forever, or immediately with no backoff.

Check yourself

2 min

Q1/3Decision

Which steps of POST /bookings should become background jobs? (Select all that apply.)

Select all that apply.

How confident are you?

Next episode: the payment provider says it will call StayHub back when a payment settles, and one callback arrives three times.

Finished?

Mark it complete to update your skill map.

Next: Webhooks: receiving and sending events over HTTP

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Retries, and the job that runs twice

Worked example

You draft the job plan, Lina checks it

Checkout takes 5.8 s and fails when the email provider is down. Move the side work out of the request without losing any of it, using the PostgreSQL StayHub already runs.

1Split now from soon
List each step and ask: does the guest need its result before "Confirmed"? Payment and the booking insert: yes. Email, invoice, channel manager sync: no, but they must happen.
Why: Only the "yes" steps belong in the request. Everything else just adds its latency and its failures to checkout.

2Record jobs with the booking

Insert the jobs in the same transaction as the booking, each with only the booking id.

await db.transaction(async (tx) => {
  const b = await tx.bookings.insert({ roomId, guestId, status: "confirmed" });
  await tx.jobs.insert({ type: "send_confirmation_email", payload: { bookingId: b.id } });
  await tx.jobs.insert({ type: "render_invoice", payload: { bookingId: b.id } });
});

Why: Lina's check: enqueueing after the commit can fail silently (a booking with no email). In one transaction, both exist or neither does.

3Let workers claim jobs safely

Two worker processes claim jobs with FOR UPDATE SKIP LOCKED and a 60-second lock. If a worker dies, the lock expires and the job is available again.

sql

UPDATE jobs SET status = 'running', locked_until = now() + interval '60 seconds', attempts = attempts + 1
WHERE id = (SELECT id FROM jobs WHERE status = 'queued' AND run_at <= now()
            ORDER BY priority DESC, run_at FOR UPDATE SKIP LOCKED LIMIT 1)
RETURNING id, type, payload, attempts;

Why: SKIP LOCKED lets several workers take different jobs without waiting on each other, and the lock timeout means no job is lost to a crash.

4Shape the retries
Transient failures (timeouts, 503) retry with : up to 1 s, 2 s, 4 s… capped at 5 minutes. Permanent ones (invalid email address) don't retry. After 8 attempts, move the job to the dead-letter table and alert.
Why: Instant retries hammer a provider that's already down. A limit stops a from looping forever.
5Make every job safe to run twice
Before sending, insert a row with a unique key confirm:<bookingId>. If it already exists, stop. Pass the same key to the email provider as an .
Why: Lina's second check: a crash after sending but before marking the job done means it will run again. Delivery is .
6Watch the oldest job
Add two alerts: the oldest queued job is older than 2 minutes, and anything lands in the dead-letter table.
Why: "10,000 jobs queued" can be fine or terrible. "Emails are 25 minutes late" is what guests feel.

Queue choice	Good at	Watch out for
Postgres jobs table	No new system; enqueue in the same transaction as the booking	Throughput ceiling, table bloat, polling load
Redis-based (BullMQ, Sidekiq)	Fast; priorities, delays, rate limits, a UI	Durability depends on Redis settings; another system to run
Managed queue (SQS, Cloud Tasks…)	Scales without ops, built-in DLQ and visibility timeout	At-least-once with duplicates; vendor limits
Event log (Kafka)	Many consumers, replay, very high throughput	Not a job queue: no per-message retry or delay

Pick the simplest option that meets your volume. Many teams run on Postgres for years.

Interactive simulation

Visual diagram

Guest browser

Booking API

PostgreSQL

Job queue

Email worker

PDF + sync workers

Email provider

Dead-letter queue

Scheduler (cron)

Monitoring

Click a component

A booking request enqueues follow-up jobs; workers process them with retries, and failures that never succeed land in a DLQ.

Analogy

In a restaurantIn your backend

In a restaurantThe waiter takes your order and leavesIn your backendThe API answers right away
In a restaurantTickets clipped on the railIn your backendJobs waiting in the queue
In a restaurantCooks grab the next ticketIn your backendWorkers take the next job
In a restaurantA dish that keeps burning, set asideIn your backendA failing job in the DLQ
In a restaurant"Make stock at 6 am" on the wallIn your backendA cron job

Where the analogy breaks: a cook who walks out takes the ticket with them. A queue puts the ticket back for another worker. That's exactly why the same job can run twice.

Check your understanding

Q1/3Reasoning

The email provider returns 503 for every attempt. Put what happens to the job in order:

1.After the fix, an engineer redrives DLQ jobs
2.Job is delayed by a growing, jittered backoff
3.Job moves to the dead-letter queue and an alert fires
4.Worker takes the job and calls the provider
5.Retries continue until the max attempt count
6.Provider returns 503; job marked failed, attempt 1

How confident are you?

Practice ladder

Step 1 · Trace

Trace an email job through an outage

Scenario Warm-up

StayHub's email job policy: retry on timeout or 5xx, wait random(0, 2^(attempt−1)) seconds (capped at 5 minutes), max 8 attempts, then the dead-letter table. Workers hold a 60-second lock per job.

At 12:00:00 a guest books. The job is inserted with the booking. The email provider is down from 11:55 to 12:03.

Trace the job: roughly when each attempt happens, whether it reaches the dead-letter table, and what the guest sees and receives.

Step 2 · Complete

Finish the invoice worker

Code Practice

Lina's invoice worker is almost done. Fill in the two gaps so the job is safe to run twice and skips bookings that no longer need an invoice.

queue.process("render_invoice", async (job) => {
  const booking = await db.bookings.find(job.payload.bookingId);
  // (A) ___ skip when there's nothing to do

  // (B) ___ make sure we never create two invoices for one booking

  const pdf = await renderInvoicePdf(booking);          // ~0.8 s of CPU
  await storage.put(`invoices/${booking.id}.pdf`, pdf);
  await db.invoices.markReady(booking.id);
});

Step 3 · Solve

Room photos take 9 seconds to upload

Design Practice

Hotel staff upload room photos in the extranet. Today POST /photos receives the image, makes 4 resized versions, strips location data, and pushes them to the CDN before answering: about 9 s per photo. Staff upload 30 at a time and sometimes get timeouts, and then upload again.

Redesign this with background jobs. Say what stays in the request, what the job carries, how staff see progress, and how you avoid duplicate photos when staff retry.

Step 4 · Decide

One queue or two before the newsletter?

Decision Stretch

Maya's first marketing newsletter goes out Friday to 48,000 past guests. It will run as jobs. Today StayHub has one jobs table and 2 workers that process about 40 jobs/s.

Karim: "Just enqueue all 48,000 Friday morning. The workers will get through them."

Write a short decision in this form: "We chose X over Y because…, accepting the risk of…". Compute what happens to a booking confirmation enqueued right after the newsletter, and say what you'd monitor.

Explain it yourself

Explain to a teammate why the confirmation email should be a background job, and what new problems that creates.

0 words