Why we're not shipping speech-to-speech yet.

Google's Gemini Live API is the most interesting thing to happen to voice AI in the last twelve months. It is a native multimodal model that takes audio in, returns audio out, and skips the conventional speech pipeline entirely. The latency win is real, somewhere between 300 and 500 ms per turn on average. We've evaluated it carefully and decided not to ship it in v1 of Vorel. This post explains the reasoning, because we expect to be asked, and because the answer is more interesting than the trade itself.

What speech-to-speech actually is

A conventional voice agent runs a chained pipeline. Audio from the caller hits a speech-to-text model (Deepgram, Whisper, AssemblyAI). The transcript hits an LLM (Claude, GPT, Gemini). The text response hits a text-to-speech model (ElevenLabs, Cartesia, OpenAI TTS). The audio response hits the caller. Three models, three vendors, three sets of latency budgets, three failure modes. This is the architecture every production voice agent in 2026 actually runs, including ours.

Speech-to-speech (S2S) collapses all three into one model. The model accepts audio as input directly, reasons over it, and emits audio as output directly. There is no transcript step in the middle, and no separate TTS at the end. Gemini's Live API is the leading 2026 production option. OpenAI's Realtime API exists but is positioned more as a developer toy than a production substrate today. Anthropic has not shipped a comparable surface. So in practice, when a vendor talks about S2S in production right now, they mean Gemini.

The mechanics look roughly like this:

chained pipeline (what we run):
  caller audio
    -> STT (Deepgram)            ~120 ms
    -> LLM (Claude / GPT)        600-4000 ms
    -> TTS (ElevenLabs)          ~180 ms
    -> caller audio

speech-to-speech (Gemini Live):
  caller audio
    -> Gemini multimodal model   ~400-3700 ms
    -> caller audio

Why we said no for v1

We want to be honest about the latency claim first, because the rest of the post is critical and we do not want to be read as dismissing the technology. The S2S latency win is not marketing. The model genuinely saves the STT step, the TTS step, and the two service hops between them. On a routine turn (short caller utterance, short reply, no tool calls), we measured 300 to 500 ms of saving against our own chained pipeline. That is not nothing. If voice latency was a flat problem where every turn shaves the same amount, S2S would be an obvious win and we would ship it. The reason we are not shipping it is everything that sits next to that latency number.

The first problem is vendor concentration. Gemini-only S2S means one supplier on the most caller-facing surface of the product. If Google rate-limits us during a traffic spike, the agent goes mute. If Google deprecates the API endpoint (which happens, frequently, in this category), we have no fallback. If Google raises prices in year two, we have no negotiating leverage. Voice infrastructure for a serious operator is a multi-decade commitment. We are not willing to bet that surface on a single API in the year it shipped.

The second problem is voice quality, which under S2S becomes the model vendor's opinion rather than the operator's choice. With a separate TTS layer, the operator picks the voice. ElevenLabs has hundreds of voices and lets you clone your own. Cartesia ships fast, expressive voices with credible emotional range. PlayHT, OpenAI, and others compete on quality and price. We can route a clinic to one voice and a service shop to another. Under S2S, you get whatever timbre Gemini produces. There is no equivalent of the curated voice library. The operator's brand voice (which is often the thing they care about most) is no longer their decision to make.

Voice quality under S2S becomes the model vendor's opinion, not the operator's choice. That is a strange thing to sign over.

The third problem is that the latency math is less favorable than the headline suggests. Our sibling post 'Voice latency is the LLM' decomposes where the seconds actually go in a production turn. The LLM thinking time runs 600 ms to 4 seconds depending on the turn. Saving 300 to 500 ms of pipeline overhead on top of that is a 7 to 30 percent improvement on the easy turns where the model returns fast, and an effectively invisible improvement on the hard turns where the model thought for two seconds. The buyer-felt latency on a p95 turn does not change much. We would be trading permanent vendor lock-in on the substrate for a margin that the caller cannot reliably perceive.

The fourth problem is diagnostic blindness. When something goes wrong in a chained pipeline, every stage is inspectable. We can pull the STT transcript and see if the model misheard a word. We can pull the LLM response and see exactly what text it produced. We can pull the TTS request and confirm the right voice was selected. We can swap any layer for a competitor in an afternoon. Under S2S, all of this is fused into one model call. The agent said the wrong thing, and we cannot easily tell whether it misheard, misreasoned, or misspoke. Production triage gets harder, and the levers for fixing a specific failure mode are gone.

What would change our mind

S2S is the right architecture eventually. The question is just when. We have three conditions we are watching for.

First, at least two production-grade S2S providers. The moment OpenAI Realtime is genuinely production-ready, or Anthropic ships Sonic, or a credible third entrant emerges, the vendor-concentration argument weakens significantly. We need a fallback. We need negotiating leverage. We need the ability to route per-tenant or per-call between providers when one is having a bad day.

Second, a router-runner abstraction inside Vorel that can A/B between S2S substrates on a per-tenant or per-call basis, the same way our existing pipeline already routes between model providers for the LLM layer. This is engineering we know how to build; we just will not build it against a single-vendor surface.

Third, parity on the features that operators actually need from a voice substrate. Reliable barge-in handling when the caller interrupts the agent mid-sentence. Emotion control credible enough to switch between a calm clinic intake and a friendly salon booking. Multilingual coverage that matches what ElevenLabs and Cartesia ship today. The 2026 generation of S2S is close on the first, mixed on the second, and incomplete on the third.

The deeper principle

There is a general rule in voice AI worth naming explicitly. The most painful vendor switch is the substrate. Swapping an LLM is a config change; we do it routinely. Swapping a TTS provider is a config change with a voice-portfolio reshuffle. Swapping the substrate of a chained pipeline is a multi-week engineering project, because every observability hook, every latency budget, every triage runbook is wired to the assumption that the layers are separate.

Pick a substrate that has more than one supplier. Otherwise you have signed up to pay whatever the supplier asks, forever, on the layer that is hardest to escape. The Gemini Live API is a real piece of engineering. We respect it. We will probably ship on it, or something like it, in 2027. We are not shipping on it as our voice substrate in 2026, because the substrate has exactly one supplier today, and that is not a bet a serious voice product should take.

Pick a substrate that has more than one supplier. The Gemini Live API does not, yet. That is the trade in one sentence.

We expect this calculus to flip within twelve to eighteen months. When it does, we will be ready to route to it, because we built our pipeline assuming the substrate is a swappable layer rather than a permanent commitment. That choice (substrate optionality) is the thing this post is really about. The S2S decision is just where it shows up first.