Episode 136 · The voice that sounded like a robot

Realtime speech-to-speech models vs cascaded pipelines

Tuesday, 11:20. StayHub's in-app voice assistant scores 3.1 out of 5: "efficient but robotic", and 950 ms per answer feels slow inside an app that's otherwise instant. A speech-to-speech model answers in 450 ms and sounds human. What do we give up?

Some AI models hear and speak directly, with no text in the middle. Faster and warmer, but harder to control.

30 min advancedRealtime speech-to-speech modelsCascaded voice pipeline: ASR → LLM → TTSRealtime audio transport: WebRTC vs WebSocketsChoosing cascaded vs speech-to-speech voice architectureVoice latency budget (end of speech → first audio)Barge-in: letting the caller interrupt

Problem
StayHub's voice concierge works, but sounds flat and still pauses noticeably. Marketing wants it to sound like a warm, quick human receptionist.
Why the simple system fails
A cascaded pipeline throws away tone of voice at the ASR step and adds three hand-offs (ASR, LLM, TTS), each with its own latency floor. Tuning gets you to about a second, rarely much lower.
Engineering concept
Speech-to-speech models process audio tokens in and audio tokens out in one model, so there is no transcription or synthesis hop. The price is losing the text checkpoint where you logged, guarded and controlled the conversation.
Technology
Realtime model APIs over WebRTC (browser/mobile) or WebSockets (server, telephony), ephemeral session tokens, server-side tool execution, transcripts as a side channel.
Trade-offs
Speed and naturalness versus control, debuggability, model choice and cost. Many production systems end up hybrid: realtime for the conversation, text-based checks around tools and sensitive actions.

The problem

2 min

Tuesday, 11:20. Two weeks into the in-app voice beta, ~2,000 sessions a day.

Maya · Product manager
App voice scores 3.1/5: "efficient but robotic". 950 ms feels slow in an instant app.
Maya · Product manager
Our test with a speech-to-speech model: about 450 ms, warm voice, 4.3/5.
Lina · Senior engineer, your mentor
You lead this one. Before you answer Maya: what would we lose?

The big idea

4 min

What the text in the middle was doing

The big idea

No text in the middle? Then control moves to the boundaries you own: who gets a session, what tools do, what tool results contain, what gets logged.

The cascade is an interpreter who writes everything down. Speech-to-speech is a bilingual host who just talks: you control what the host is handed and allowed to do.

How it works

5 min

A takes audio in and produces audio out, hearing and making tone, with no text step you control. The cascade keeps separate ASR, LLM and TTS stages with your text in between.

One realtime session, end to end

1
Guest signs in
Our backend checks who the guest is, as usual.
2
Mint a short-lived token
An ephemeral token: the backend creates a session with instructions and allowed tools baked in; the app gets a 60 s token, never the API key.
3
Open WebRTC
carries live audio over UDP with jitter buffering and ; a lost packet doesn't stall the rest.
4
Tools go through us
Each tool call: policy check, validated arguments, 1.5 s timeout, and a result filtered to fields safe to say aloud.
5
Log what's true
Tool calls and arguments are the record of actions; transcripts are approximate side outputs.

Watch a decision being made

4 min

Worked example

You wire realtime voice into the app (Lina checks)

Bring speech-to-speech to ~2,000 sessions a day, keeping the cascade's controls: no leaked keys, no unsafe actions, nothing sensitive spoken, a record of what happened.

1List what the text turn protected
Exact logs, output checks, model routing. For each, find where it can live without text.
Why: Lina: "Most teams discover this list after an incident."
2Pick the transport, keep the key home
WebRTC from the app; WebSockets only server-to-server. A 60-second token from our backend.
Why: Over TCP one lost packet stalls everything behind it (). A key in the app gets extracted.

3Filter every tool result

Return only the fields the answer needs:

function toSpeakable(b: Booking) {
  return { hotel: b.hotelName, checkIn: b.checkIn, nights: b.nights,
           guests: b.guests.length };   // no names, IDs or card data
}

Why: Whatever enters the session can be spoken.

4Keep a way back
A feature flag switches a hotel back to the cascade; compare ratings, cost and incidents weekly.
Why: Voice models change fast; a flag makes the model a setting, not a rewrite.

In real life

3 min

Two well-known approaches

ChatGPT voice mode: audio end to end

OpenAI described GPT-4o as handling audio natively, in one model
Fast, hears tone; fewer places to inspect each turn as text

Fits when conversation quality is the product.

Classic assistants like Alexa: a pipeline

Publicly described as recognition, then understanding, then a synthesised reply
Each stage tested, logged and swappable

Fits when predictability matters most.

What goes wrong

3 min

Shipping the provider key in the app

Anyone can extract it and run sessions on your bill, with any instructions.

Instead

Short-lived, pre-configured session tokens from your backend.

Treating realtime transcripts as the record

They can differ from what was heard.

Instead

Log tool calls and arguments; label transcripts approximate.

Choosing speech-to-speech for speed on phone lines

Carrier latency and endpointing dominate; you pay in control and cost.

Instead

Measure end to end on real calls first.

Trade-offs

3 min

Option	Buys	Costs
Cascade	Text checkpoint every turn, any LLM	Higher latency floor, flat tone
Speech-to-speech	Lowest latency, real tone	No owned text; ~3× per minute
Hybrid	Warmth plus checkpoints per call type	Two architectures to run

Monday, 9:30. The passport fix is live. Roll realtime out to all hotels?

Karim · CTO
Realtime is ~480 € a day versus 160 € for the cascade. And we just had an incident.
Maya · Product manager
Ratings went from 3.1 to 4.3. Voice users book spa and restaurant slots twice as often.
Karim · CTO
Then prove it covers 320 € a day.

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own

Which architecture each voice flow uses: realtime where feel is the product, the cascade where you need exact text.
What every tool returns into a session: only fields the agent may say aloud.
How sessions are authorised (short-lived tokens) and how you switch back (a flag per flow).

Tell Claude Code

Move the app's voice concierge to a realtime speech-to-speech session behind a flag; the cascade in `services/voice/` stays as fallback. Plan first. WebRTC with a short-lived token minted by our backend, scoped to the guest and to get_booking_summary and check_availability; tools run server-side and check the guest owns the booking; results pass an allowlist of speakable fields (no passport, ID, email, phone, payment). Test that a companion's passport number never enters the session.

Check before you accept

The exact JSON one tool result sends for a two-guest booking: every field fine to hear aloud.
Only the session token appears in the app bundle, never the API key.
Flipping the flag off in staging falls back to the cascade without a release.

Red flags

Personal data "fixed" with a prompt line while tool results still contain it.
The long-lived API key in the mobile app.
Audio over a plain WebSocket from phones on mobile networks.

Check yourself

2 min

Q1/3Scenario

In a realtime voice session, it is safe to return the full guest profile as a tool result because the system prompt tells the model not to read out personal data.

How confident are you?

Next episode: the voice assistant, the chat and the booking flow each run their own agent loop, and Karim wants one runtime for all of them before the next incident.

Finished?

Mark it complete to update your skill map.

Next: The agent runtime: production architecture around the model

Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.

Technical deep dive

Wiring realtime safely

Worked example

You wire realtime voice into the app (Lina checks)

Bring the speech-to-speech model into the StayHub app for ~2,000 sessions a day, keeping the controls the cascade gave us: no leaked keys, no unsafe actions, nothing sensitive spoken, and a record of what happened.

1List what the text turn protected
Write down what the cascade's per-turn text did: exact logs, output checks, model routing. For each, find where it can live without text.
Why: You can't keep a control you haven't named. Lina: "Good. Most teams discover this list after an incident."
2Pick the transport
WebRTC from the app (UDP, jitter buffer, echo cancellation). WebSockets only server-to-server.
Why: Over TCP, one lost packet stalls everything behind it (). On mobile networks that's an audible freeze.
3Keep the key on the server
The backend authenticates the guest and mints a 60-second session token with the instructions and that guest's allowed tools baked in.
Why: A provider key in the app can be extracted and used on our bill, with any instructions. Baking the config in also stops the app from changing the prompt.
4Check at every tool boundary
Policy check per call, schema-validated arguments, a 1.5 s timeout, and toSpeakable() that returns only the fields the answer needs.
ts
```
function toSpeakable(b: Booking) {
  return { hotel: b.hotelName, checkIn: b.checkIn, nights: b.nights,
           guests: b.guests.length };   // no names, IDs or card data
}
```
Why: Whatever enters the session can be spoken. The tool boundary is now our only per-action checkpoint.
5Log the truth and keep a way back
Log tool calls and arguments (redacted) as the record of actions; label transcripts as approximate. Add a feature flag to switch a hotel back to the cascade, and compare ratings, cost and incidents weekly.
Why: Realtime transcripts can differ from what was said. And voice models change fast: a flag makes the model a setting, not a rewrite.

Dimension	Cascaded (ASR → LLM → TTS)	Speech-to-speech realtime
Time to first audio	~0.7–1.2 s tuned (illustrative)	Often a few hundred ms lower (illustrative)
Naturalness	Tone lost at ASR; generic TTS prosody	Hears and produces tone, emotion, hesitation
Control	Text on every turn: check, rewrite, block	Only around tools and via instructions
Model choice	Any text LLM; mix ASR/TTS vendors	Few realtime models; harder to route
Observability	Exact input and output text per turn	Session events; transcripts approximate
Cost	Sum of three services, often cheaper	Audio tokens usually priced higher

Not a ranking. Each row matters more or less depending on your product and its risks.

Interactive simulation

Visual diagram

Guest app (browser)

Phone caller

Session server

Telephony bridge

Realtime S2S model

Tool executor

Traces + audio log

Click a component

A realtime voice architecture for app and phone callers. Your backend still owns tokens, tools and logs.

Analogy

At a hotel deskIn a voice agent

At a hotel deskA note-taker writes what the guest saysIn a voice agentSpeech recognition (ASR)
At a hotel deskA manager reads the note and repliesIn a voice agentThe text model (LLM)
At a hotel deskAn announcer reads the reply aloudIn a voice agentText-to-speech (TTS)
At a hotel deskNotes filed, checked, stopped if wrongIn a voice agentText checkpoint in the cascade
At a hotel deskOne fluent receptionist who just answersIn a voice agentA realtime speech-to-speech model

Where the analogy breaks: the realtime model does leave notes (transcripts and tool calls). You just can't review a reply before it's spoken, so your safety checks move to the tools around it.

Check your understanding

Q1/3Decision

A bank wants a phone agent that must block any spoken disclosure of account numbers and keep an exact text record of each turn for auditors. Which architecture fits best?

How confident are you?

Practice ladder

Step 1 · Trace

Trace a realtime session's tool calls

Scenario Warm-up

A guest's realtime session produces these events. The policy allows get_booking, check_spa_slots and book_spa (with confirmation) for this guest; refund_payment is not allowed.

1 tool_call get_booking        { bookingId: "B-4472" }
2 tool_call check_spa_slots    { date: "2026-10-12" }
3 tool_call book_spa           { slot: "17:00", confirmed: false }
4 tool_call refund_payment     { bookingId: "B-4472", amount: 120 }
5 tool_call check_spa_slots    { date: "next friday" }

For each call, say what our tool handler does and what goes back into the session.

Step 2 · Complete

Finish the session endpoint

Code Practice

Lina's session endpoint has three gaps. Fill them in and say what each prevents.

export async function createVoiceSession(req: Request) {
  const guest = ___;                              // (1)
  const session = await realtime.createSession({
    instructions: conciergePrompt(guest),
    tools: ___,                                   // (2)
    expiresInSeconds: ___,                        // (3)
  });
  await audit.log("voice_session_created", { guestId: guest.id });
  return Response.json({ token: session.clientSecret });
}

Step 3 · Solve

Should the phone line go realtime too?

Estimation Practice

Karim: "Realtime is faster. Switch the phone line too."

Phone facts from StayHub Voice: - Current cascade (the tuned phone pipeline), measured from end of speech: endpointing 450 ms, network 60 ms both ways, ASR 80, LLM first clause 250, TTS 120 → about 960 ms. - A realtime model would replace ASR + LLM + TTS (450 ms) with about 300 ms to first audio. - ~8,000 calls a day, ~3 min each. Cascade ~0.04 €/min, realtime ~0.12 €/min (illustrative). - Phone callers are anonymous; 1 in 50 calls involves a payment dispute.

Estimate the gain and the cost, and recommend.

Step 4 · Decide

Spa upgrades with payment, by voice

Decision Stretch

Maya wants guests to buy spa upgrades (40–180 €) by voice in the app, charged to the card on file. The realtime assistant already handles spa bookings without payment.

Write a short decision: "We chose X over Y because…, accepting the risk of…". Consider: doing it all in the realtime session, routing the payment step to the cascade, or a confirmation outside voice.

Explain it yourself

Explain to a CTO when you would use a realtime speech-to-speech model instead of a cascaded pipeline, and what you would build around it.

0 words