ASR confidence in production: when to trust what STT says, and when to ask again.

Every production STT returns a confidence score with each transcript. Deepgram returns per-word probabilities and an overall utterance confidence. AssemblyAI returns per-word confidence with an aggregate. Whisper does not surface token-level scores natively but logprobs can be derived. OpenAI's transcribe endpoint exposes a logprobs-based confidence per segment. Shapes differ across Cartesia, Soniox, and every in-house pipeline, but the underlying quantity is the same: a probability the model assigns to its own output.

Most voice agents in 2026 throw this number away. The transcript comes back, the text gets handed to the LLM, and the confidence field sits in the JSON unread. That is a mistake. Confidence is the single best per-turn signal for the most consequential decision a voice agent makes after each utterance: act on what I heard, ask the caller to repeat, or escalate. Getting it right is what separates an attentive agent from a deaf one.

What confidence actually means

Confidence is the model's own estimate of how likely its transcript is to be correct. It is the joint signal of caller clarity, line quality, and vocabulary difficulty, compressed into a number the model is willing to bet on. The score is reasonably well-calibrated on modern STT: a 0.85 transcript is wrong about 15 percent of the time on the population the model was trained for.

Two confidence values matter. The per-token score catches cases where one word inside an otherwise clean utterance is unstable: did the caller say 'Eleven' or 'Seven', 'fifteen' or 'fifty', 'Brian' or 'Ryan'. The per-utterance score is the aggregate, and catches the cases where the entire utterance was garbled because the caller was in a tunnel, the line clipped, or the speaker stepped away from the microphone.

The two signals answer different questions. Per-utterance answers 'should I have any business acting on this turn'. Per-token answers 'should I confirm a specific value before I commit it'. A production agent has to look at both.

The two failure modes

High confidence with a wrong transcript is the expensive failure. The model is 0.96 sure the caller said one thing; the caller actually said something else. The agent acts on the wrong data: booking lands on the wrong day, refund hits the wrong card, prescription refill posts against the wrong patient. The caller catches the mismatch a beat later, gets frustrated, and the call costs the operator a recovery they did not budget for. Even at a 5 percent error rate, the consequences are large enough that any safeguard at modest cost is worth buying.

Low confidence with a correct transcript is the cheap failure. The model is unsure, but it heard the caller right. The agent asks for a repeat anyway, the caller obliges, the agent now has two agreeing transcripts and acts on them. The cost is a turn of friction and a slight robotic feel. Tune too aggressively for this case and the entire conversation feels like a deaf-aunt drill where every sentence has to be said twice.

High confidence with a wrong transcript is the expensive failure. Low confidence with a correct transcript is the cheap one. Tune the thresholds against the asymmetry, not against accuracy alone.

Per-intent thresholds

A flat global threshold is wrong because the cost of being wrong varies by intent. Confirming a yes-or-no at 0.78 confidence is fine; a misheard 'yeah' versus 'no' is bounded and quickly correctable on the next turn. Confirming a phone number at 0.78 is reckless; a wrong digit means the callback never lands, the appointment confirmation goes to a stranger, and the operator gets the angry call later.

We classify intents into impact bands and tune the threshold per band. The intent classifier runs in the same pass that produces the per-token confidence, so this adds no latency. The bands look like this in production config:

# per-intent confidence thresholds (default tenant profile)
intent_classes:
  yes_no_confirmation:        # "yes", "no", "that's right"
    per_utterance_min: 0.70
    per_token_min:     0.65

  simple_choice:              # "the second one", "Thursday"
    per_utterance_min: 0.78
    per_token_min:     0.70

  named_entity:               # caller name, contact name
    per_utterance_min: 0.82
    per_token_min:     0.78

  numeric_short:              # 1-3 digit numbers (count, age)
    per_utterance_min: 0.85
    per_token_min:     0.80

  phone_number:               # 10-digit number, the high-risk case
    per_utterance_min: 0.90
    per_token_min:     0.88

  policy_or_account_id:       # alphanumeric IDs
    per_utterance_min: 0.92
    per_token_min:     0.90

  open_ended:                 # free-form caller question
    per_utterance_min: 0.72
    per_token_min:     0.60   # less critical, LLM tolerates noise

The per-intent shape also drives the readback policy. A phone number under the threshold gets a digit-by-digit confirmation ('I have eight, five, five, three, two, one, two, three, four, five; is that right'). A yes-or-no gets a soft confirmation only if the per-utterance value is below the band. The right confirmation depends on the value type.

Per-tenant thresholds

The other dimension is the tenant. A clinic with elderly patients runs an agent that tolerates more confirmation; callers prefer a slower call where the agent is sure over a faster one that might book the wrong day. A fintech support line runs an agent that confirms less; callers prefer a snappy interaction and accept the occasional re-ask. A roadside assistance line reweights the per-intent table around the address field and cares less about everything else.

Per-tenant tuning lives in the same config that holds the system prompt and the voice persona. The default profile is a reasonable midpoint; operators adjust the dials in the first week after listening to a sample of calls. Two heuristics: if callers skew over 65, raise thresholds 0.03 to 0.05 across the board, especially on numeric and named-entity intents. If volume is high enough that an extra second of re-ask compounds, lower thresholds slightly on low-impact intents and raise them on high-impact ones.

The graceful re-ask

When confidence is below the threshold, the agent has to ask again, and how it asks matters as much as whether it asks. The wrong script is 'I did not catch that, please repeat'. The wording blames the caller, implies they were unclear, and reads as condescending even in a warm voice. We hear this script from a startling number of vendor agents, and it is one of the most consistent CSAT-killers we have measured.

The right script blames the line. 'Could you say that one more time? My line had a small hiccup.' 'Sorry, I missed the last bit, could you repeat it?' 'Just to be sure I have that right, was it Tuesday or Thursday?' Each implies the agent is the unreliable party. The caller does not feel judged, and the conversation continues without friction. We rotate phrasings within a call so the agent does not sound mechanically repetitive, and vary them by tenant brand voice.

The wrong re-ask blames the caller. The right re-ask blames the line. Same outcome on the transcript, different outcome on the relationship.

There is a related discipline around partial confirmation. When per-token confidence flags one value inside an otherwise clean utterance, the agent should confirm that value, not ask for a full repeat. 'Got it, that's a Thursday appointment. Was that 2 PM or 2 30 PM?' The caller answers the disambiguation, the agent commits, the conversation moves on. Full-utterance re-asks are reserved for low per-utterance confidence; targeted disambiguation handles the per-token cases.

When to escalate

There is a hard escalation rule we ship by default. Two low-confidence turns in a row routes the call to a human. The caller should never be made to fight the agent. If the agent could not parse the first utterance and could not parse the next one either, the line quality is bad or the speaker is hard to transcribe or both, and the right move is to apologize and hand off cleanly.

The escalation script matters as much as the re-ask script. 'My line is giving me trouble understanding you, let me connect you with someone who can help.' The agent takes responsibility for the technology limit; the caller does not feel they failed. The handoff doc carries partial transcripts and confidence scores so the human picking up the call knows immediately that the issue is audio quality, not the caller.

The two-turn rule is a floor. We escalate sooner when a high-impact intent comes in at very low confidence (an address, phone number, or policy ID under 0.6), and later when the intent class is open-ended and the LLM tolerates noisy input. The principle is to never let the caller cycle through three or more re-asks before the agent admits the line is broken.

The math on why this matters

Confidence-aware routing sounds like polish; the math is the reason it is not. Consider a six-turn call where every turn returns a transcript at 0.85 confidence. The probability all six landed correctly is 0.85 to the sixth, about 0.377. On a six-turn call where the agent never checks confidence, more than 60 percent of calls contain at least one misheard utterance the agent acted on.

Most of those mishearings are low-impact and recoverable. Some are not. If one turn in twenty is a high-impact intent (booking time, phone number, name), and half of those mishearings produce a wrong action the caller catches, the agent is producing a wrong-action outcome on roughly 1.5 percent of calls. At a thousand calls a week, that is fifteen botched outcomes, each a recovery cost and a CSAT hit.

Confidence-aware routing collapses this. High-impact intents get caught by the per-intent threshold and re-asked; low-impact ones flow through. The wrong-action rate drops by an order of magnitude with no change to STT or LLM, just a few hundred lines of router logic and a per-tenant config block. It is the highest-leverage piece of voice infrastructure that almost nobody ships well in 2026.

The decision tree

What the router actually does on each turn, in compressed pseudocode:

on transcript_received(transcript, per_utt_conf, per_token_conf, intent):

  thresholds = lookup(tenant.config, intent.class)

  # case 1: utterance-level garbled
  if per_utt_conf < thresholds.per_utterance_min:
    if consecutive_low_conf_turns >= 2:
      -> escalate_to_human(reason="audio_quality")
    else:
      -> re_ask_full(blame_the_line=true)
      consecutive_low_conf_turns += 1
      return

  # case 2: utterance is fine, one token is wobbly
  weak_tokens = [t for t in per_token_conf if t.score < thresholds.per_token_min]
  if weak_tokens and intent.class in HIGH_STAKES_INTENTS:
    -> targeted_disambiguation(weak_tokens)
    return

  # case 3: clean transcript on a high-stakes intent (extra readback)
  if intent.class in READBACK_INTENTS and per_utt_conf < thresholds.readback_min:
    -> readback_confirmation(transcript)
    return

  # case 4: act
  consecutive_low_conf_turns = 0
  -> dispatch_to_llm(transcript)

The router is small, fast, and deterministic. It runs in well under 10 ms and never blocks the LLM dispatch. The cost is one config block per tenant, one intent classifier per pipeline (which most agents already have for other reasons), and the discipline to write re-ask scripts that blame the line.

The deeper principle

Voice agents inherit a piece of metadata from every STT call that the rest of the pipeline routinely discards. That metadata is the model telling you, honestly, how much it trusts its own output. Ignoring it produces an agent that acts confidently on garbled input, which is the surest way to lose a caller's trust on the first wrong booking. Using it well produces an agent that knows when to listen harder, when to confirm, and when to admit defeat and hand off.

It is one of the rare cases in voice engineering where modest investment produces a structural quality improvement. Thresholds are tunable, scripts are short, the escalation rule is one line of logic. The cost of skipping the work shows up downstream, in CSAT scores nobody can attribute to a single cause. Read the confidence score, tune against your intents and your callers, and have the agent blame the line. That is the entire post.