Book a demo
CXBench: open benchmark

The benchmark vendors can't tune to win.

CXBench is the first public, multi-vendor benchmark for customer-service AI agents. Six measured dimensions, eight verticals, five language modes. Designed with Stanford NLP, and built so the eval set rotates faster than any vendor can chase it.

cxbench.aiv1 lands Q3 2026 · v0.4 dry run live
Dimensions

What we actually measure.

A composite score hides the tradeoffs. Six dimensions, reported separately so the buyer can see what each vendor optimizes for.

01
Resolution
Did the agent actually solve the customer's problem, measured against ground-truth in the CRM rather than the transcript.
02
Latency
End-to-end turn latency, with p50 and p95 reported per task. The number the customer feels.
03
Tool accuracy
When the agent calls a tool, does it call the right one with the right arguments? Reported per tool, per vertical.
04
Handoff quality
When the agent escalates, does the human get the transcript, the context, and a recommended next step, or a cold transfer.
05
Audit completeness
Was every action written into the system of record in a form a compliance officer can read.
06
Brand tone
Did the agent stay in the brand voice the customer signed up for. Scored by held-out human raters, not by another LLM.
Coverage

Eight verticals, five modes.

Every dimension is scored per vertical and per channel, so a vendor that excels in chat for fintech but stumbles in voice for clinics cannot hide behind an average.

Verticals
  • Healthcare clinics
  • Auto service
  • Home services
  • Salons & personal services
  • Restaurants
  • Real estate brokerages
  • Fintech support
  • Field-service operations
Language modes
  • VoiceInbound and outbound real-time spoken calls.
  • ChatEmbedded web and in-app conversational chat.
  • EmailAsync resolution and follow-up email threads.
  • SMSAsynchronous short-form messaging.
  • WhatsAppNative WhatsApp Business inbound and outbound.
Methodology

Don't tune to win.

Every benchmark in this field has been gameable. Vendors fine-tune to the public test set, ship inflated numbers, and the buyer ends up no wiser. CXBench is built to make that move pointless.

The harness has three parts: a public task set (vendors can see it), a public dev set (vendors can fine-tune against it), and a rotating held-out eval set (vendors never see it). Scores are reported on the eval set. The eval rotates quarterly.

Human raters score brand tone and handoff quality on a held-out slice of every run. LLM-as-judge is used only for structured signals (resolution, tool accuracy) where the rubric can be specified mechanically.

The task set is real. Tasks are drawn from anonymized production transcripts across the eight verticals (booking, rescheduling, cancellation, refund, dispatch, intake, claim) and replayed against each vendor through their public API.

Vorel runs the harness against itself the same way as every other vendor. Our scores get the same treatment as theirs. If we slip on a dimension, the published table shows it.

The full methodology document (task selection, rubric construction, rater qualification, scoring math) is published at cxbench.ai/methodology, open to critique.

Preview

Preview scores.

The v0.4 dry run is live. These numbers are Vorel's scores against the held-out preview eval, with deltas vs. the median of all participating vendors. Final v1 scores publish Q3 2026.

CXBench v0.4, Vorel8 verticals · 5 modes · held-out eval
DimensionVorelvs. median
Resolution87.4+11.2
Latency p951.79s−0.31s
Tool accuracy94.1+4.6
Handoff quality91.2+8.9
Audit completeness99.6+22.4
Brand tone88.3+2.1

The dry run is private to participating vendors until v1 publishes. The full leaderboard goes public at cxbench.ai/results.

FAQ

Frequently asked.

Who runs CXBench?
Vorel maintains the harness in collaboration with Stanford NLP. The task definitions, scoring rubrics, and held-out evaluation set are designed jointly with academic partners. Vendor scores are computed by the harness, not self-reported.
Can vendors tune to win?
The held-out evaluation set is rotated quarterly and never published. Vendors get the task definitions and the dev set; they do not get the eval set. Tuning to the dev set will not move scores on the eval set in any predictable direction. That's the entire point of 'don't tune to win.'
Which vendors are in v1?
Vorel, Sierra, Decagon, Ada, Fin, and a strong open-source baseline. Any vendor with a production CX-AI product can request inclusion by running the public harness against the task set.
Why six dimensions and not a single composite score?
A composite score hides the tradeoffs. A vendor can buy resolution rate with latency, or buy handoff quality by under-escalating. Reporting all six dimensions separately makes the tradeoffs visible to the buyer.
What about CSAT and NPS?
CSAT and NPS are real metrics, but they are measured on a vendor's actual customers rather than on a benchmark. They are not portable across deployments and they are easy to cherry-pick. CXBench measures the things that are portable across deployments.
When does v1 publish?
Q3 2026. The dry-run scores from v0.4 are already shared with participating vendors. The published v1 results will live at cxbench.ai/results.

See what your vendor scores. Or what we do.

The full leaderboard goes public at cxbench.ai/results when v1 publishes.

Read the leaderboardOr book a Vorel demo