Voice latency is the LLM. Everything else is a rounding error.

Every voice-AI vendor in 2026 publishes a 'sub-second latency' number. Every demo lands in under a second. Every production deployment is over two seconds on the worst turn (sometimes over three), and the customer notices. The gap between the headline number and the felt experience is the single most expensive lie in this category.

We've been running voice agents in production for long enough to decompose where the seconds actually go. The answer turns out to be boring: the LLM is the entire cost. Almost everything else (STT, TTS, the network hops between services) combines to under 250 ms on a good pipeline. The LLM thinking time, by itself, is 600 ms to 4 seconds depending on the turn. If you optimize anything other than the model layer, you are sweeping pennies off a pool of dollars.

The decomposition

Take a single production turn. Caller stops talking, agent starts replying. The pipeline runs roughly like this:

caller stops talking
  endpointing detection      120-200 ms   (Deepgram + VAD)
  STT final transcript        80-160 ms
  context assembly             5-15 ms
  LLM thinking time         600-4000 ms   (this dominates)
  tool calls (if any)        80-400 ms    (one round-trip per tool)
  TTS first audio chunk      120-240 ms
  network egress              20-60 ms
caller hears first word

The headline turn (model returns a 30-token plain answer) lands around 900 ms. The pathological turn (model reasons over 8 tools, calls 3, reasons again, then replies) lands around 3500 ms. The variance is almost entirely the model. Everything else is a roundoff.

Why this matters for the buyer

Voice-AI vendors love to talk about their 'optimized pipeline': fewer hops, lower-overhead protocol, custom STT, on-prem model serving. Some of this is real engineering. None of it moves the dial. Saving 80 ms of network hop on a 3-second LLM turn is a one-and-a-half percent improvement that the customer cannot feel.

The headline you should ask for is p95, not average. p50 is the easy turn (short reply, no tools). p95 is the turn where the model thought hard. The gap between p50 and p95 on a well-built voice agent is the gap between 'felt great' and 'I think the AI broke.' If a vendor publishes only p50, they are hiding the tail.

The headline you should ask for is p95, not average. A vendor who only publishes p50 is hiding the tail.

How to actually reduce voice latency

Knowing the LLM dominates changes the playbook entirely. The interventions that move the needle, in rough order of impact:

Pin a smaller model on routine turns. Most voice turns are short replies with one or zero tool calls. A 4-8B fine-tuned model handles those at sub-300 ms thinking time. You only need the big model for the long-tail turns where reasoning actually helps. A serious vendor routes per-turn based on the conversation shape, not per-deployment based on the contract.

Parallelize tool calls. Out of the box, agentic frameworks fire tools sequentially. When the model wants to look up the patient and check the schedule, the right move is to fire both at once. A typical turn that calls 2-3 tools shrinks from 600 ms to 300 ms once you do this correctly. Most vendors do not.

Stream the first audible word as soon as the model produces it. Don't wait for the full response. The customer hears 'I can help with that, let me check' while the model is still deciding what to do. This is mostly a perception fix, not a real latency fix, but perception is what the customer rates.

Skip the LLM entirely when you can. A booking confirmation that says 'You're booked for Thursday at 2:15 PM' does not require a frontier model to compose. A small template fired deterministically from the tool result is faster, cheaper, and indistinguishable to the caller.

What does not move the dial

We hear vendors talk about all of these. None of them are wrong, but none of them are the bottleneck:

Self-hosted inference. Worth doing for sovereignty and cost. Does not improve latency materially against a well-routed managed inference provider. The hosted providers are already running the same hardware.

Custom protocols on top of WebRTC. The network hop between services is single-digit ms on a co-located cluster. Even doubling that to 10 ms makes no perceptible difference next to a 1.5 s LLM turn.

Speech-to-speech (S2S) end-to-end models. In theory they collapse the pipeline. In practice the current generation has worse tool-calling and worse handoff than the pipeline-with-STT-and-TTS approach. Sierra called this out in their own blog, and we agree. S2S will be the right answer in 18 months, not today.

What this means for CXBench

When we designed the CXBench latency rubric, we made two decisions in light of this:

First, we report p95 as the headline number. p50 is published as a sub-metric but it is not the score. The buyer-relevant question is how the agent performs on the hard turns, not the easy ones.

Second, we score latency separately from resolution. A vendor cannot trade resolution for latency without it showing up in the table. The vendors who do trade (by short-circuiting the reasoning, by refusing to call the second tool) will look fast and unhelpful, side by side.

Latency is a real engineering problem, and it deserves to be measured honestly. The single biggest favor a vendor can do for the category is publish the methodology behind their numbers. We do, and we expect the rest of the field will follow once buyers start asking for the same.