Start free trial
CXBench: open benchmark

The benchmark vendors can't tune to win.

CXBench is a public, multi-vendor benchmark for customer-service AI agents, now in development. Six measured dimensions, eight verticals, five language modes, and an eval set built to rotate faster than any vendor can chase it.

cxbench.aiIn development · methodology preview
Dimensions

What we actually measure.

A composite score hides the tradeoffs. Six dimensions, reported separately so the buyer can see what each vendor optimizes for.

01
Resolution
Did the agent actually solve the customer's problem, measured against ground-truth in the CRM rather than the transcript.
02
Latency
End-to-end turn latency, with p50 and p95 reported per task. The number the customer feels.
03
Tool accuracy
When the agent calls a tool, does it call the right one with the right arguments? Reported per tool, per vertical.
04
Handoff quality
When the agent escalates, does the human get the transcript, the context, and a recommended next step, or a cold transfer.
05
Audit completeness
Was every action written into the system of record in a form a compliance officer can read.
06
Brand tone
Did the agent stay in the brand voice the customer signed up for. Scored by held-out human raters, not by another LLM.
Coverage

Eight verticals, five modes.

Every dimension is scored per vertical and per channel, so a vendor that excels in chat for fintech but stumbles in voice for clinics cannot hide behind an average.

Verticals
  • Healthcare clinics
  • Auto service
  • Home services
  • Salons & personal services
  • Restaurants
  • Real estate brokerages
  • Fintech support
  • Field-service operations
Language modes
  • VoiceInbound and outbound real-time spoken calls.
  • ChatEmbedded web and in-app conversational chat.
  • EmailAsync resolution and follow-up email threads.
  • SMSAsynchronous short-form messaging.
  • WhatsAppNative WhatsApp Business inbound and outbound.
Methodology

Don't tune to win.

Every benchmark in this field has been gameable. Vendors fine-tune to the public test set, ship inflated numbers, and the buyer ends up no wiser. CXBench is built to make that move pointless.

The harness has three parts: a public task set (vendors can see it), a public dev set (vendors can fine-tune against it), and a rotating held-out eval set (vendors never see it). Scores are reported on the eval set. The eval rotates quarterly.

Human raters score brand tone and handoff quality on a held-out slice of every run. LLM-as-judge is used only for structured signals (resolution, tool accuracy) where the rubric can be specified mechanically.

The task set is real. Tasks are drawn from anonymized production transcripts across the eight verticals (booking, rescheduling, cancellation, refund, dispatch, intake, claim) and replayed against each vendor through their public API.

Vorel runs the harness against itself the same way as every other vendor. Our scores get the same treatment as theirs. If we slip on a dimension, the published table shows it.

The full methodology document (task selection, rubric construction, rater qualification, scoring math) is published at cxbench.ai/methodology, open to critique.

Preview

Preview scores.

This is the report format every participating vendor receives: six dimensions scored separately, with deltas against the participating-vendor median. The numbers below are an illustrative sample of the format, not measured results — published scores land with v1.

Report format (illustrative sample)8 verticals · 5 modes · held-out eval
DimensionVorelvs. median
Resolution87.4+11.2
Latency p951.79s−0.31s
Tool accuracy94.1+4.6
Handoff quality91.2+8.9
Audit completeness99.6+22.4
Brand tone88.3+2.1

Sample numbers shown for format illustration only. The live leaderboard goes public at cxbench.ai/results when v1 ships.

FAQ

Frequently asked.

Who runs CXBench?
Vorel maintains the harness and is recruiting independent academic partners to co-own the task definitions, scoring rubrics, and held-out evaluation set before v1 publishes. Vendor scores are computed by the harness, not self-reported.
Can vendors tune to win?
The held-out evaluation set is rotated quarterly and never published. Vendors get the task definitions and the dev set; they do not get the eval set. Tuning to the dev set will not move scores on the eval set in any predictable direction. That's the entire point of 'don't tune to win.'
Which vendors are in v1?
At launch: Vorel and a strong open-source baseline, with an open, public invitation to every major vendor. Any vendor with a production CX-AI product can request inclusion by running the public harness against the task set.
Why six dimensions and not a single composite score?
A composite score hides the tradeoffs. A vendor can buy resolution rate with latency, or buy handoff quality by under-escalating. Reporting all six dimensions separately makes the tradeoffs visible to the buyer.
What about CSAT and NPS?
CSAT and NPS are real metrics, but they are measured on a vendor's actual customers rather than on a benchmark. They are not portable across deployments and they are easy to cherry-pick. CXBench measures the things that are portable across deployments.
When does v1 publish?
CXBench is in development; the harness and dev set publish first, then the scored leaderboard. Published results will live at cxbench.ai/results.

See what your vendor scores. Or what we do.

The full leaderboard goes public at cxbench.ai/results when v1 publishes.

Read the leaderboardOr book a Vorel demo