Skip the LLM: templated short-circuits for the easy 30% of voice turns.

There is a category of voice turn that does not require a frontier model. The caller asks what time the clinic opens on Saturday. The tool returns 9 to 1. The agent says 'We're open Saturday from 9 to 1.' The model's contribution to that turn is zero. It read a JSON value out of a tool result, wrapped it in a sentence frame, and handed it to the TTS. Any of that work could have been done by a 12-line format string and a deterministic dispatcher.

In production we see this pattern on roughly 30 percent of turns. Booking confirmations. Reschedule confirmations. Hours-of-operation lookups. 'You're now connected, one moment.' 'Let me check that for you' filler while a long-running tool call resolves in the background. None of these require reasoning. All of them get routed through the LLM in a naive build, because the naive build does not have a router that knows the difference.

We do have that router. This post is about the architecture, the heuristic we use to decide when a turn qualifies, and the numbers we get on the other side.

The turns that do not need reasoning

The category is narrower than it first sounds. A turn qualifies for a templated short-circuit when three things are simultaneously true. First, the intent is high-confidence (a booking confirmation after a successful booking tool call is high-confidence; an open-ended customer question is not). Second, the tool result, if there is one, has a clean, predictable shape (a booking object with a date and time is clean; a free-form note from the CRM is not). Third, the speakable response is deterministic from the inputs (the booking confirmation says the date, the time, and asks if anything else is needed; the inputs fully determine the output).

The turns that fit this description, in our deployments, are concentrated in a handful of buckets. Confirmation turns after a successful booking, reschedule, or cancellation. Lookup turns where the tool returns a single canonical value (hours of operation, the address of a location, whether a particular insurance is accepted). Connector turns that fire while a longer tool call is running ("let me look that up for you, one moment"). Handoff turns where the agent is announcing a transfer ("connecting you with the front desk now"). The vast majority of caller-facing language in these turns is sentence-frame, not content.

The turns that do not fit are the ones where the model is actually doing work. Explaining why a claim was denied. Walking a caller through three options and helping them pick. Handling a complaint where tone matters. Disambiguating an under-specified request. These are real reasoning problems, the kind a templated turn would mangle, and we leave them on the model.

The architecture

The router sits between the tool layer and what would normally be the next LLM call. When a tool returns, the router inspects the tool name, the result shape, and the conversation-state intent classifier (a small, fast classifier that already runs in our pipeline for other reasons). If the combination matches a registered short-circuit, the router formats a template directly from the tool result and ships it to TTS. The LLM never sees the turn.

A template is a tagged format string with slot variables. It lives in the same tenant config that holds the system prompt, the voice persona, and the operator-specific phrasing rules. A booking-confirmation template for a dental clinic looks roughly like this:

template: booking_confirmed
trigger:
  tool: book_appointment
  result_shape: { status: "ok", appointment: { date, time, provider } }
  intent_confidence: >= 0.92
format: |
  You're booked with {appointment.provider} on {appointment.date | dayOfWeek}
  at {appointment.time | speakable}. Is there anything else?
voice: clinic_warm_v3
fallback: llm  # if format/parse fails, kick the turn back to the model

The flow on a templated turn looks like this:

caller turn
  -> STT                       ~120 ms
  -> tool call (book_appt)      ~80 ms
  -> router:
       tool name      = book_appointment            (match)
       result.status  = "ok"                        (match)
       intent.conf    = 0.96                        (>= 0.92)
       -> SHORT-CIRCUIT
  -> template format              ~5 ms
  -> TTS first audio chunk      ~150 ms
  -> caller hears first word
  --------------------------------------
  total                          ~355 ms

For comparison, the same turn through the model lands around 1.5 seconds in the best case and over 2 seconds on the tail. The caller hears the templated response three to four times faster. The inference bill on the templated turn is zero.

On confirmation turns the caller cannot tell the difference, because there is no difference worth telling. The model would have produced the same sentence.

Where templates are wrong

We want to be honest about what templates lose. They are stiff. They do not pick up the subtle phrasing variations a good model produces turn-to-turn (the small acknowledgments, the natural reorderings, the warmth that comes from the model occasionally choosing a slightly different sentence frame). On a routine booking confirmation, the caller cannot hear the stiffness, because there is no acoustic-grade difference between 'You're booked for Thursday at 2 PM' generated by Claude and the same sentence generated by a format string. On a turn where the caller is upset, or confused, or trying to negotiate, the stiffness shows up immediately. A template that says 'I understand your frustration. Can you tell me more?' six times in a row sounds exactly like what it is.

So the heuristic is conservative on purpose. We only short-circuit when intent confidence is high (we hold the bar at 0.92 in production), when the tool result shape is one we have explicitly registered, and when the template has a fallback to the LLM if the format step encounters anything unexpected. A missing slot, an unrecognized status code, a tool error: any of these kicks the turn back to the model. The router is a fast path, not a substitute path.

The other failure mode is staleness. Operator personas drift. A clinic decides they want their agent to say "we look forward to seeing you" at the end of every booking confirmation. If that line lives in the system prompt and the booking confirmation is templated, the template will not pick up the change and the agent will sound inconsistent. We mitigate this by colocating templates with the system prompt in the same tenant config and versioning them together: every persona deploy regenerates both. A template that drifts from the prompt is a deploy bug, not a runtime mystery.

What it costs and what it saves

The latency math is the obvious win. A templated turn lands around 355 ms. A model turn through our chained pipeline lands around 1.5 seconds on the easy case and 2 to 3 seconds on the hard case. On the 30 percent of turns we route through templates, the caller hears the response three to four times faster. Across a full call, that compresses the dead air enough that the conversational rhythm feels closer to a human exchange, which matters more than the raw numbers suggest.

The cost math is the less obvious win. Every templated turn pays zero inference cost. At 30 percent of turns templated, the LLM bill drops 30 percent against a naive build, before any other optimization. For a voice deployment running 50,000 turns a month, this is meaningful money, and it is money we pass through to the customer rather than recapture as margin. The arithmetic is friendly: every templated turn is faster, cheaper, and (within the conservative routing rules) indistinguishable from the model turn it replaced.

There is a second-order benefit worth naming. Templated turns are inspectable in a way that model turns are not. The format string is in the config; the inputs are in the tool log; the output is deterministic from the two. When a clinic asks 'why did the agent say that to my patient,' we can hand them the exact format string and the exact tool result, and the answer is unambiguous. Model-generated turns are auditable in the looser sense (we have the prompt and the response), but they are not deterministic. Templates are.

The deeper principle

Not every voice turn is a reasoning problem. A lot of them are formatting problems. A lot of them are acknowledgment problems. A lot of them are filler-while-something-else-runs problems. The naive architecture treats every turn the same way (route through the model, wait for tokens, ship to TTS) because the naive architecture cannot tell the difference. The wins come from architectures that can.

The biggest cost and latency wins in voice AI come from recognizing which turns are not reasoning problems in the first place.

We expect this pattern to generalize beyond the buckets we have registered so far. Small-talk acknowledgments at the start of a call. Goodbye turns at the end. Hold-music announcements when a queue gets long. Any turn where the speakable response is determined by the inputs and not by reasoning is a candidate. The router is cheap to extend; the templates are cheap to author; the fallback to the LLM is the safety net that keeps the conservative posture honest.

If you are building voice AI today and you are routing every turn through the model, you are paying for reasoning you did not need on 30 percent of your traffic. The fix is not exotic. It is a small router, a handful of templates, and a willingness to be honest about which turns are the model's job and which turns are not.