Skip to content
Episode 135 · Three seconds of silence on the phone

The voice agent pipeline: microphone to speaker in under a second

Friday, 23:40. StayHub's new phone concierge answers every question correctly, after 3.5 seconds of silence. At Hotel Mar Azul, 31% of callers hang up before the second answer.

How a phone agent hears you, thinks and talks back in under a second, and why every stage fights for milliseconds.

30 min advancedCascaded voice pipeline: ASR → LLM → TTSTurn detection and endpointing (VAD)Barge-in: letting the caller interruptVoice latency budget (end of speech → first audio)Telephony integration for voice agents (PSTN, SIP, codecs)Latency budgets across hops
  1. Problem

    Guests call the hotel at 23:00 to ask about rooms, late check-in and parking. Nobody answers, and a chat widget does not help someone driving.

  2. Why the simple system fails

    Gluing a chatbot to speech-to-text and text-to-speech in sequence gives 3–4 seconds of silence per turn, cuts people off mid-sentence and cannot be interrupted. Callers say “Hello? Are you there?” and hang up.

  3. Engineering concept

    A voice agent is a streaming pipeline with a latency budget: every stage starts before the previous one finishes, turn detection decides when to answer, and barge-in lets the human take the floor back.

  4. Technology

    VAD, streaming ASR, an LLM with tool calling, streaming TTS, WebRTC or a telephony bridge (SIP), filler phrases and echo cancellation.

  5. Trade-offs

    Faster turn detection cuts people off; bigger models reason better but answer later; streaming everything adds engineering complexity and makes failures partial instead of clean.

01

The problem

2 min
Friday, 23:40. First weekend of the StayHub Voice pilot at Hotel Mar Azul.
  1. Maya · Product manager
    The phone concierge answers correctly, after 3.5 s of silence each time. 31% of callers hang up before the second answer.
  2. You · Engineer at StayHub
    Then the model is too slow. Switch to a faster one?
  3. Lina · Senior engineer, your mentor
    Guess where the time goes first.
02

The big idea

4 min
Where 3.5 seconds of silence come from
The big idea

In voice, the caller hears your architecture: stream every stage so they overlap, and measure from the moment the caller stops speaking.

Same runners, same speed, shorter race: the next runner starts as soon as the baton is in reach, not when the last one crosses the finish.

03

How it works

5 min

A cascaded voice agent is → model → , inside one long-lived session per call. The number callers feel is time to first audio: from when they stop talking to the first sound of the answer.

One phone turn: batch (v1) vs streaming (v2)
  1. 1
    Detect the end of the turn

    Endpointing decides the caller has finished, after a short silence spotted by a . v1 waits 800 ms; v2 450 ms, longer if the sentence looks unfinished.

  2. 2
    Speech to text

    v1 sends the whole clip (550 ms). Streaming ASR transcribes while they talk, leaving an ~80 ms final pass.

  3. 3
    The model replies

    v1 waits for the full reply (1.6 s). v2 streams a small model and hands over the first clause after ~250 ms.

  4. 4
    Text to speech, then play

    v1 synthesises the whole answer (500 ms). Streaming TTS plays the first clause after ~120 ms.

  5. 5
    Cancel on interruption

    Barge-in: when the caller interrupts, the session cancels speech and in-flight requests at once.

04

Watch a decision being made

4 min
Worked example

Lina gets a turn under one second

Get typical turns from about 3,570 ms to about 1 s without making answers worse.

  1. 1Trace every turn by stage

    Endpoint, ASR final, LLM first clause, tool, TTS first audio, network, for every turn.

    text
    endpoint     800 ms
asr_final    550 ms   (batch)
llm_full    1600 ms   (large model)
tts_full     500 ms
network      120 ms
first_audio 3570 ms

    Why: Without per-stage numbers, "the model is slow" is a guess. The trace shows four slices, not one.

  2. 2Stream all three stages

    Streaming ASR; a small model streaming clauses; TTS per clause.

    Why: The caller needs the first sentence, not the paragraph.

  3. 3Mask the tools

    When the model calls the PMS (300–900 ms), say "Let me check that" at once; cache availability 60 s; time out at 1,500 ms with a spoken fallback.

    Why: A 900 ms lookup without a filler is nearly two seconds of dead air.

  4. 4Tune endpointing, measure again

    800 → 450 ms, extended when the transcript looks unfinished. New total: about 960 ms.

    Why: Endpointing is now the biggest slice, but shorter would cut off people who pause to think.

The whole thing in one line: Trace, fix the biggest slice, measure again: the same answers now start after 0.96 s instead of 3.57 s.
05

In real life

3 min
Two voice products, two ways to take turns

Amazon Alexa: explicit turns

  • A wake word starts each request
  • Mostly short commands; simple, predictable turn-taking

Fits devices with short commands.

Google Duplex: natural phone calls

  • Publicly demoed in 2018 booking a salon and a restaurant
  • Natural turn-taking, described as working in narrow domains only

Fits narrow phone tasks, at high engineering cost.

06

What goes wrong

3 min
Very short endpointing to look fast

At 200 ms the agent answers while people pause to think.

Instead

~450 ms, extended when the sentence looks unfinished.

Silence while a tool runs

Callers think the line dropped.

Instead

Filler phrase, cache, timeout with a spoken fallback.

Trusting transcripts of names and numbers

Phone ASR confuses "fifteen" and "fifty".

Instead

Read codes back, send links by SMS, keep a human transfer.

07

Trade-offs

3 min
ChoiceBuysCosts
Short endpointing (250–350 ms)Snappy short answersCuts off slow speakers
Long endpointing (600–900 ms)Rarely interruptsA pause before every turn
Large model every turnBest reasoning2–4× slower first token
Small model + routingFast, cheapNeeds a router and evals
Wednesday, 16:00. Reviewing a week of v2 calls.
  1. Maya · Product manager
    Guests love the speed. Drop endpointing to 300 ms?
  2. Omar · On-call SRE
    At 450 ms, 6% of turns are cut off. At 300 ms it was 14%, worse for older callers.
  3. Maya · Product manager
    Most questions are short. Those would be snappier.
08

Your job as tech lead

4 min

Claude Code can write the code for this. You still own the decisions, and you have to know what good looks like to accept or reject its work.

Decisions you own
  • The latency budget per turn, from when the caller stops speaking: about 1 s to first audio, with a number per stage.
  • The endpointing silence and the model that answers on the phone.
  • What happens during slow tools, and when barge-in may be turned on (only with echo cancellation).
Tell Claude Code
Make StayHub Voice for Hotel Mar Azul answer faster; turns take ~3,570 ms to first audio and 31% hang up. Plan first, then change `services/voice/` one stage at a time: streaming ASR and TTS; endpointing 450 ms, longer mid-sentence; the small model streams its first clause; filler plus a 1,500 ms timeout on check_availability. Barge-in stays off. Add per-turn stage traces; report p50/p95 time to first audio on the 50 recorded calls.
Check before you accept
  • Stage numbers in the trace add up to the total, and you can name the biggest slice.
  • Before/after measured on the same 50 calls, from when the caller stops speaking.
  • A slowed PMS double gives a filler then a spoken fallback, never silence.
Red flags
  • Endpointing dropped to 150 ms because the chart looks great.
  • Barge-in turned on without echo cancellation.
  • Latency measured from when the LLM was called.
09

Check yourself

2 min
Q1/3Reasoning
A turn has: 400 ms endpointing silence, 80 ms ASR final pass, 300 ms LLM time to first token, 100 ms to generate the first clause, 120 ms TTS first chunk, and 40 ms network each way. What is the time to first audio, in ms?
ms
How confident are you?

Next episode: StayHub's mobile app wants a voice button, and a new speech-to-speech model promises to skip text altogether. Is the middle step worth keeping?

Finished?

Mark it complete to update your skill map.

Next: Realtime speech-to-speech models vs cascaded pipelines
Go deeper (optional)You don't need this to understand the lesson. Open it when a project needs the details: Technical deep dive, Interactive simulation, Visual diagram, Analogy, Check your understanding, Practice ladder, Explain it yourself.
10

Technical deep dive

Getting one turn under a second
Worked example

Lina gets a turn under one second

StayHub Voice takes about 3,570 ms from the end of the caller's question to the first sound of the answer, and 31% of callers hang up. Get typical turns under about 1 s, without making answers worse.

  1. 1Trace every turn by stage

    Record endpoint time, ASR final, LLM first token, tool duration, TTS first audio and network, for every turn of every call.

    text
    turn 3  call 8812  "Is parking free on Friday?"
endpoint     800 ms
asr_final    550 ms   (batch, whole clip)
llm_full    1600 ms   (large model, full reply)
tts_full     500 ms
network      120 ms
first_audio 3570 ms

    Why: Without per-stage numbers, "the model is slow" is a guess. The trace shows four slices worth attacking, not one.

  2. 2Stream speech to text

    Switch to streaming ASR: audio frames are pushed as they arrive, so only a short final pass (~80 ms) remains when the caller stops.

    Why: Transcribing during speech hides most of the ASR time inside the caller's own talking.

  3. 3Stream the model, clause by clause

    Stream tokens, cut them into clauses at , . ? !, and hand each clause to TTS. Use a small, fast model by default (~250 ms to the first clause), routing complaints to a larger one.

    Why: The caller needs the first sentence, not the whole paragraph. A smaller model is 2–4× faster to the first token.

  4. 4Stream text to speech

    Synthesise per clause and play the first audio chunk as soon as it exists (~120 ms).

    Why: Waiting for the full audio file wastes the time we just saved upstream.

  5. 5Tune endpointing, then measure again

    Lower the silence threshold from 800 to 450 ms, and wait longer when the transcript looks unfinished ("I'd like to book for…"). New trace: about 960 ms.

    Why: Endpointing is now the biggest slice, but cutting it further would interrupt people who pause to think. Measuring again shows where the next fix is.

The whole thing in one line: Lina traced each turn, made every stage stream so they overlap, picked a faster model for the first clause, and tuned endpointing: the same answers now start after about 0.96 s instead of 3.57 s.
StageBatch (v1)Streaming, tuned (v2)Main lever
Endpointing silence800 ms300–500 msShorter threshold + incomplete-sentence detection
Network to server40–80 ms20–40 msRegion close to callers / telephony provider
ASR400–700 ms (whole clip)~80 ms (final pass)Streaming ASR during speech
LLM to first clause1,500+ ms (full reply)150–400 msSmaller or routed model, stream tokens, short prompts
Tool call300–1,000 msmasked by filler / cacheCaching, timeouts, prefetch
TTS to first audio500+ ms (full reply)~100–150 msStreaming TTS per clause
Illustrative order-of-magnitude numbers. Measure your own providers; they vary with load, language and region.
11

Interactive simulation

12

Visual diagram

Click a component

A cascaded hotel voice concierge. Click a component, or follow one caller turn.

13

Analogy

  • A live interpreterNoticing someone starts speakingA voice agentVAD spots speech
  • A live interpreterUnderstanding words as they comeA voice agentStreaming speech recognition
  • A live interpreterTelling a breath from a finished pointA voice agentEndpointing
  • A live interpreterStarting a sentence before planning allA voice agentStreaming text-to-speech
  • A live interpreterStopping mid-word when interruptedA voice agentBarge-in

Where the analogy breaks: an interpreter just translates. A voice agent also has to act, like checking rooms or booking, and those tool calls can stall the conversation in a way an interpreter never would.

14

Check your understanding

Q1/3Reasoning
Order what happens in one streaming turn, starting when the caller begins to speak:
  1. 1.VAD detects silence after the last word
  2. 2.Endpointing threshold passes: the turn is over
  3. 3.Caller hears the first audio
  4. 4.ASR emits partial transcripts while the caller talks
  5. 5.TTS synthesises that clause
  6. 6.LLM streams the first clause of its reply
How confident are you?
15

Practice ladder

Step 1 · Trace

Read a turn trace

Debugging Warm-up

A v2 turn trace from a caller asking "Do you have a room on Saturday?":

endpoint          450 ms
asr_final          90 ms
llm_first_clause  260 ms   -> tool_use: check_availability
filler_tts        110 ms   "Let me check that for you."
tool              870 ms
llm_after_tool    240 ms
tts_first_audio   120 ms
network            60 ms
  1. When does the caller hear something (the filler)?
  2. When do they hear the actual answer?
  3. Which slice would you attack next, and how?
Step 2 · Complete

Finish the barge-in handler

Code Practice

When the caller starts speaking while the agent talks, the agent must stop at once. Lina's handler has two missing lines. Fill them in and explain what each one prevents.

onAudio(frame: Frame) {
  const isSpeech = this.vad.isSpeech(frame);   // echo already cancelled
  if (isSpeech) this.speechMs += 20; else this.speechMs = 0;
  if (this.speaking && this.speechMs >= 150) this.bargeIn();
  this.asr.push(frame);
}

private bargeIn() {
  ___                                   // (1)
  ___                                   // (2)
  this.trace.mark("barge_in");
}
Step 3 · Solve

"Fifteen" or "fifty"?

Design Practice

Omar: "Last week the concierge added 9 arrival notes with the wrong date or booking code. One guest said 'arriving the fifteenth' and the note says the 50th… which the PMS rejected, so the guest's late arrival was never recorded."

Design how StayHub Voice should handle names, dates and booking codes so these errors are caught on the call.

Step 4 · Decide

Large model on every call for the luxury hotels?

Decision Stretch

Karim: "Our 5-star partners want the 'best' concierge. Let's use the large model on every turn for them."

The facts: - Small model: ~250 ms to the first clause. Large model: ~800 ms. - On StayHub's eval set of 400 hotel calls, the small model handles 91% of turns correctly; the large one 96%. Most of the gap is in complaints and multi-room requests (about 10% of turns). - Large model cost per call minute is about 4× the small one.

Write a short decision: "We chose X over Y because…, accepting the risk of…".

16

Explain it yourself

Explain to a product manager why the hotel voice agent paused for three seconds in the prototype, and what the production design does differently.

0 words