Skip to content

Voice AI

Microphone → model → speaker, fast enough to feel like a conversation.

speech recognitiontext-to-speechturn-takingrealtime multimodal

Lessons

  1. 32.1The voice agent pipeline: microphone to speaker in under a secondHow a phone agent hears you, thinks and talks back in under a second, and why every stage fights for milliseconds. 30 min Voice agent latency budget
  2. 32.2Realtime speech-to-speech models vs cascaded pipelinesSome AI models hear and speak directly, with no text in the middle. Faster and warmer, but harder to control. 30 min Voice agent latency budget

On the job: try it on your own project

Pick a phone or spoken interaction your users or team have today. Sketch the voice pipeline for it: speech to text, model, text to speech, and turn-taking. Ask Claude Code to estimate the latency of each step and the cost per minute. Write a one-page note on whether voice is worth building now, and what latency would make it feel natural.

Can you do this now?

0/4. Anything you can't tick yet: reopen that lesson's practice ladder.