Streaming the first audible word. Why perceived latency beats actual latency in voice.

There are two latency numbers in a voice agent and only one of them shows up on a customer CSAT survey. The first is server-side end-to-end latency: the time from the caller's last audio frame to the agent's last audio frame. The second is caller-perceived latency: the time from the caller's last audio frame to the agent's first audible word. These numbers used to be roughly the same. They no longer are, and the gap between them is the single biggest perceived-latency win in voice AI.

This post is the engineering pair to our broader latency post. That post argued that LLM thinking time dominates the chained pipeline. This post argues that even after you have shrunk LLM thinking time as far as it will go, there is one more lever the buyer can hear: stream the first audible word out the door before the model has finished thinking. Done right, the caller hears the agent begin to speak in 350 to 500ms. The full reply still lands in 1.5 to 2 seconds. The caller does not notice the second number, because they stop counting once they hear a voice.

Two latencies, only one matters to the buyer

Server-side end-to-end tells the engineer whether the pipeline is healthy. Caller-perceived first-word latency tells the operator whether the call sounds good. The engineer can produce a clean, fast end-to-end number and still ship an agent that the caller experiences as slow, because the caller is sitting in silence waiting for something to happen. The same 1.8-second turn feels like a fluent reply when the agent begins speaking at 400ms and feels like a buffering chat widget when the agent begins speaking at 1.8 seconds. The information content is identical. The lived experience is not.

The same 1.8-second turn feels like a fluent reply when the agent begins speaking at 400ms and feels like a buffering chat widget when the agent begins speaking at 1.8 seconds.

The architecture

The naive build collects the entire LLM response, hands it to the TTS, and ships the resulting audio. The TTS does not start synthesizing until the model has stopped generating. The caller hears nothing until both have finished. This is what most vendor pipelines do out of the box, and it is what makes their demos feel snappy on short replies and dead on long ones.

The streaming-first build watches the LLM token stream and waits for the first sentence boundary, or the first six to ten tokens, whichever lands first. The moment that prefix is in hand, it is piped to the TTS as a partial. The TTS begins synthesizing immediately. The caller starts hearing the agent speak while the model is still producing the rest of the reply. As more tokens arrive, they are appended to the TTS stream and the audio continues without a seam.

caller stops speaking
  t = 0 ms       endpointing detection (VAD + Deepgram)
  t = 130 ms     STT final transcript
  t = 145 ms     context assembly
  t = 145 ms     LLM thinking begins
  t = 280 ms     first sentence boundary in token stream
                   ("Sure, let me take a look at that for you.")
  t = 290 ms     prefix piped to TTS streaming endpoint
  t = 425 ms     first audio chunk back from TTS
  t = 445 ms     first audio frame egresses to Twilio
  -------------------------------------------------------
  t = 445 ms     CALLER HEARS FIRST WORD     <- streaming
                 (model is still generating the rest)

  t = 1800 ms    LLM finishes generating
  t = 1850 ms    last audio chunk synthesized
  t = 1880 ms    CALLER HEARS LAST WORD

compare to the non-streaming build:

  t = 0 ms       caller stops speaking
  t = 1800 ms    LLM finishes generating
  t = 1980 ms    full TTS synthesis complete
  t = 2000 ms    CALLER HEARS FIRST WORD     <- non-streaming
  t = 3450 ms    CALLER HEARS LAST WORD

The total reply duration is roughly the same. The first-word latency drops from 2000ms to 445ms. The caller is hearing the agent speak in under half a second instead of waiting a full two seconds in silence. Server-side end-to-end is unchanged. The CSAT-relevant number drops by a factor of four.

The constraint: which sentences are safe to commit to

Streaming the first sentence works only if the first sentence is safe to commit to before the model knows the rest of the answer. This is the part of the architecture that most teams get wrong on the first try.

Acknowledgement openers are safe. "Sure, let me take a look at that for you." "One moment while I pull up your account." "Of course, I can help with that." These prefixes do not depend on the answer the model is about to give. They commit to a posture (helpful, working on it) and fit naturally in front of essentially any reply the model produces after them. The router can ship them the moment they appear without risk of the model later reversing course.

Direct answers are not safe. "The earliest available slot is..." starts speaking before the model has finished reasoning about whether there is an available slot at all. "Your appointment is confirmed for..." starts speaking before the booking tool has fired. If the booking fails, the agent has lied to the caller in the first 400ms of the turn.

The router distinguishes these intent classes with a small, fast classifier on the user utterance plus the conversation state. High-confidence acknowledgement intents get the streaming-first path. Open-ended directive intents wait for the full reply before TTS begins. When the classifier is wrong, the fallback is to swallow the streamed prefix and ship the full reply. The caller hears nothing strange; the reply just lands 200ms slower than the streaming case.

Acknowledgement openers commit only to a posture, not to an answer. That is what makes them safe to stream before the model has finished thinking.

The TTS constraint

Not every TTS provider supports this. The streaming-first architecture requires a TTS that can accept partial input, begin synthesizing on the partial, and append more text mid-stream without restarting the synthesis. ElevenLabs ships this as their streaming API and it works in production. Cartesia ships a comparable surface. OpenAI's TTS endpoint takes the full string and synthesizes a complete file; it is not built for incremental appends. PlayHT has a streaming mode but the join between partials is sometimes audible at the seam.

This is the one place where TTS vendor choice is load-bearing on perceived latency, and most buyers do not realize it. A vendor running a non-streaming TTS cannot ship this architecture at all. They can claim sub-second numbers on the marketing site because the LLM is fast and the TTS is fast, but their caller still waits for the full reply before they hear anything. The numbers look fine in isolation; the experience is the older, slower one.

The race: tool calls after speech has started

The hardest failure mode is the one where the model opens its mouth with an acknowledgement, then decides mid-turn that it needs to call a tool. The caller hears "Sure, let me take a look at that for you" land in 400ms. The model emits a tool call. The tool round-trip takes 200ms. The model resumes, finishes generating the answer, and the TTS appends the rest of the reply. From the caller's perspective, the agent paused briefly between sentences. The pause is usually under a second and reads as natural conversational rhythm.

The hostile case is the one where the tool fails or returns something that changes the reply entirely. The cheap path is to let the acknowledgement finish naturally and have the next sentence carry the change ("actually, looks like that slot is no longer free; the next one I have is Thursday at three"). The expensive path is to interrupt the TTS, drop the queued audio, and have the agent reorient. The cheap path is almost always right, because the caller does not perceive the acknowledgement as a commitment to a specific answer. They perceive it as a commitment to engagement, and that commitment is satisfied as long as the next sentence delivers a real response.

Interruption is only unavoidable when the agent has already begun speaking a direct factual claim that turns out to be wrong. This is why the router does not stream direct-answer prefixes in the first place; the retraction sounds far worse than a half-second pause.

Report both numbers, lead with the caller-facing one

We track server-side end-to-end (the pipeline-health number) and caller-perceived first-word latency (the experience number). The operator dashboard surfaces the caller-perceived number because that is the one the operator can connect to a CSAT score on a specific call.

On CXBench, the public benchmark we are shipping in Q3, both will be reported separately. Vendors that ship streaming-first will show a large gap between their server-side number and their caller-perceived number, and the gap is itself a quality signal. The other risk is the inverse: a vendor with aggressive streaming and a slow LLM can land a 400ms first-word number while delivering a 4-second full reply. The caller hears the agent speak fast, then sits through a long awkward pause. Reporting both numbers prominently is the only way to keep that failure mode visible.

The deeper principle

Voice-AI engineering has spent two years optimizing the wrong number. End-to-end latency is the number the engineer can measure cleanly, the number that fits on a dashboard, the number a vendor can publish without context. Caller-perceived latency is the number that drives the experience. Streaming-first is the architecture that recognizes the difference and ships against the second number, even when it costs the engineer something on the first.

The architecture that wins ships against the number the caller actually feels, even when the dashboard wants to measure something else.

If your first-word latency is anywhere near your end-to-end latency, the caller is sitting in silence longer than they need to. The fix is a streaming TTS, a classifier for which prefixes are safe to commit to, and the discipline to report both numbers when the buyer asks. The architecture is about two weeks of engineering, and the CSAT lift is the largest perceived-latency win available in the stack today.