Barge-in: how the agent shuts up the moment the caller starts talking.

Listen to two humans on the phone. They interrupt each other constantly, not rudely, just naturally. One starts answering before the other finishes the question. The caller jumps in with a correction. The agent stops mid-word, takes the new input, and adjusts. This is how human dialogue works, and it is the single most underestimated thing about building a voice agent that feels alive.

Picture a robotic voice agent instead. The caller hears it say 'The earliest available time is Thursday at two...' and cuts in with 'I need Tuesday'. The bad agent finishes its sentence anyway. '...thirty PM. Would you like to book that?' By the time it stops, the caller has said the same thing twice and started pushing zero. Three seconds later the agent offers Thursday again, because nothing in its state model registered the interrupt as new information.

There are four engineering pieces: detection, interrupt, partial-transcript handling, and state recovery. The tuning between them is where the polish lives.

Detection. Why this is harder than it looks.

The intuitive answer is to run voice-activity detection (VAD) on the caller's audio while the agent is talking, and trigger an interrupt the moment the VAD threshold trips. In a quiet room this works fine. In production, callers are not in quiet rooms. They are in cars, next to TVs, walking past coworkers. There is a dog. There is a baby. There is the spouse asking what time dinner is.

A naive VAD trips on any of those, the agent shuts up, the caller never said anything, the agent waits for a transcript that never finalizes, and after a second of silence the agent asks the caller to repeat themselves for no reason. Compounded across a call, this is more annoying than the robotic-finishes-its-sentence failure. False positives are worse than false negatives, because they convert a working call into a stalled one.

False positives are worse than false negatives. A caller whose agent shuts up in response to a dog is a different kind of broken.

The fix is a layered detection model. VAD energy asks whether there is sound on the caller channel. A fast speech classifier asks whether it sounds like speech (versus a TV, a dog, a kitchen). A held-back commit window asks whether the sound has lasted long enough to be a real interrupt attempt and not a cough. A self-echo gate asks whether what we think is the caller might just be the agent leaking back through a bad headset.

Only when all four signals agree does the pipeline declare a barge-in. The math is conservative on purpose: we would rather talk past a sneeze than shut up for one. The commit window is the dial operators feel most. A clinic phone tree where callers are calm can run an aggressive 80ms window. A roadside service line where callers are in a moving car needs 180ms or false positives ruin every call.

The decisive interrupt. Stop now, not soon.

Once the pipeline declares a barge-in, the agent has to stop immediately. Not at the end of the current sentence, not at the end of the current word. Mid-syllable if that is where the interrupt landed. A delay of even 200ms past the detection point reads as the agent talking over the caller, which is what barge-in handling is supposed to prevent.

Stopping cleanly is three actions. Cancel the TTS stream at the model. Flush the audio buffer between the TTS service and the telephony bridge (otherwise buffered audio keeps playing for another 300-800ms). Record what the agent had actually streamed up to the cut, so it can reason about what the caller plausibly heard. The third part is the one most pipelines skip, and it is the one that produces the dumb-agent behavior people quote.

Here is the canonical sequence on a clean interrupt:

caller:    [silence]                                    [starts talking]
                                                          |
agent:     "The earliest available time is Thursday at|two thir-"
                                                          |     ^
                                                          |     (TTS cancel + buffer flush)
                                                          |
detection:                                                v
           VAD energy            ---------- on ---------->|
           speech classifier     ---------- on ---------->|
           commit window         --- 120ms hold ------->  |
           self-echo gate        ---------- clear ------->|
                                                          v
                                                       BARGE-IN
                                                          |
                                                          v
agent state:  said_so_far = "The earliest available time is Thursday at two thir-"
              tts_cancelled_at = T+1.42s
              waiting_for: STT final on interrupt audio
                                                          |
caller:                                                   "I need Tuesday"
                                                          |
STT:                                                      [partial: "I need Tu..."]
                                                          [partial: "I need Tuesday"]
                                                          [final:   "I need Tuesday"]
                                                          |
                                                          v
agent reasons with:
  said_so_far:  "...Thursday at two thir-"   (caller heard Thursday)
  caller_said:  "I need Tuesday"
  next:         offer Tuesday slot, do not re-pitch Thursday

The two annotations that matter are tts_cancelled_at and said_so_far. The first is a timing primitive. The second is the conversational primitive most LLM systems do not natively track. The agent needs to know, on the next turn, that it had already said the word "Thursday" out loud before it got cut off. Otherwise the recovery turn opens with "Yes, Thursday at two thirty works, would you like to book?" which is precisely the response that produces the rage-quit and the call to a human.

Partial transcripts. Waiting for STT to close.

When the agent cuts off, the caller's interrupt utterance is in flight. The speech-to-text model is streaming partials and has not yet emitted a final. The agent has a choice: act on the latest partial (fast, lossy, occasionally wrong), or wait for the STT final (correct, sometimes slow). The wrong answer is to wait until silence, then take whatever the STT happens to have, because real callers do not pause cleanly between thoughts.

The pattern we use is a two-stage hold. A tight endpointing window after the last partial token (200-350ms of caller silence); if it closes, we accept the latest partial as final. A hard 1.5s timeout regardless of partial state catches a hung STT or a caller who said one word and went quiet. Neither stage triggers the LLM until the transcript is stable, because feeding a half-finished sentence to the model produces a half-finished response.

State recovery. Telling the model what the agent already said.

This is the piece most LLM systems do not natively support, and it is the piece that decides whether the recovery turn feels human. Most chat-LLM APIs track the conversation as a sequence of complete turns. Barge-in violates that. The agent said a partial thing. The user interrupted before the partial finished. The next turn has to reason about a half-spoken utterance the model never finished generating.

You have to model this explicitly. The conversation state holds the partial assistant utterance, the timestamp of the cut, and a flag that the previous turn was barge-in interrupted. The next prompt includes a structured block that says, in effect, "The previous assistant message was cut off after the following content: <partial>. The user interrupted with: <transcript>. Continue accordingly." The model handles this well once you give it the right shape. Without the shape, it confidently produces a reply that pretends the cut never happened.

Most LLM systems track conversations as a sequence of complete turns. Barge-in violates that. Model the partial utterance explicitly, or the agent will pretend the cut never happened.

The benefit of getting this right is not subtle. A recovered turn that acknowledges the interrupt ("Got it, let me check Tuesday instead") feels human in a way nothing else in the voice pipeline can produce. A recovered turn that ignores it ("Thursday at two thirty, would you like to confirm?") is the fastest way to make the caller ask for a human. We score recovery quality separately in our voice QA, because it correlates more strongly with caller satisfaction than raw latency on easy turns.

Every value in the barge-in stack trades responsiveness for stability, and the published vendor latency numbers never tell you where the dials are set. The two that matter most: the commit window (80ms is aggressive and false-positives on sneezes; 180ms is conservative and occasionally talks over a real interrupt; most vertical defaults sit between 100ms and 140ms), and the endpointing window after the interrupt (200ms feels snappy but clips callers mid-thought; 500ms feels patient but laggy; elderly callers in healthcare run longer, young callers in a service request shorter). Two more are easy to miss. Self-echo gate sensitivity has to know both audio streams and run cross-correlation, which telephony bridges that do not expose the outbound stream cannot do well. Said-so-far granularity has to be syllable-level on long numbers and times, because "Thursday at two thir-" needs to know the caller heard "two", not "two thirty".

Why this matters more than the latency numbers

Most vendor voice demos in 2026 are recorded in clean rooms with scripted prompts; the interrupt scenarios are themselves scripted, with the caller waiting for a natural pause and 'interrupting' on cue. This is turn-taking with the production value of a podcast, not barge-in. The question to ask in a live demo is simple. Pick a long agent turn (hours-of-operation read, policy explanation). Cut in mid-word with new information that contradicts the agent. See if it stops cleanly, acknowledges the new info, and recovers without restating the old plan.

Our sibling post on voice latency argues that the LLM dominates the time budget. That is true. But once latency is good enough that the agent feels fast, the next thing the caller notices is whether the agent is listening. A 900ms reply that talks past the caller feels worse than a 1.4s reply that stops the moment the caller interrupts. Barge-in is the rhythm of the conversation, and the rhythm is what makes the agent feel alive.

We treat barge-in as a first-class production surface, with its own telemetry (interrupt rate per call, recovery latency, false-positive rate, said-so-far hit rate), its own per-vertical defaults, and its own QA replay corpus of recorded interrupts pulled from anonymized production traffic. When a vendor ships a voice agent that cannot handle interrupts, it is because nobody on their side built that surface and nobody on the buyer's side knew to look for it.

The pieces of a voice agent that make it feel human are the pieces that match how humans behave on the phone, not the pieces that look good in a sales deck. Real humans interrupt. Real humans speak with background noise. An agent that handles both feels like a person. An agent that handles neither is a phone tree with more compute behind it.