Why we built a benchmark we don't control.

Every benchmark in this field has been gameable. Vendors fine-tune to the public test set, ship inflated numbers, and the buyer ends up no wiser. We watched this happen across LLM leaderboards, code-eval suites, and the early conversational-AI benchmarks. The cycle is predictable: a benchmark gets adopted, the scores inflate, the benchmark loses credibility, a new one launches, the cycle repeats.

We did not want to launch into a market where the benchmark was the marketing surface. So we built one we could not win by gaming, and we made it possible for our competitors to participate on the same terms.

We built a benchmark we could not win by gaming, and made it possible for competitors to participate on the same terms.

The mechanism: a rotating eval set

CXBench has three parts. The first is a public task definition. It describes the tasks the agent has to perform: book an appointment, file a claim, resolve a return, triage an after-hours emergency. The definition is public so any vendor can build against it.

The second is a public dev set, sample inputs vendors can use to develop and self-evaluate. The dev set is intentionally large enough to be useful and intentionally not the scoring set.

The third is the eval set: held-out, rotated quarterly, never published. Scores are computed on the eval set. Vendors get the task definitions and the dev set. They do not get the eval set. Tuning to the dev set will not move scores on the eval set in any predictable direction. That is the entire point.

The eval set is drawn from anonymized production transcripts across eight verticals (clinics, auto service, home services, salons, restaurants, real estate, fintech support, field-service ops). The transcripts are pre-processed to strip PII, then replayed against each vendor through their public API. The vendors do not know which transcripts they were tested on.

Six dimensions, not one composite

A composite score hides tradeoffs. A vendor can buy resolution rate with latency. They can buy handoff quality by under-escalating. They can buy audit completeness by writing meaningless rows. We report six dimensions separately so the tradeoffs are visible:

Resolution: did the agent solve the customer's actual problem, measured against ground-truth in the CRM rather than the transcript. Latency: p50 and p95 reported per task. Tool accuracy: when the agent calls a tool, does it call the right one with the right arguments. Handoff quality: when the agent escalates, does the human get the transcript, the context, and a recommended next step. Audit completeness: was every action written into the system of record in a form a compliance officer can read. Brand tone: did the agent stay in the brand voice the customer signed up for.

Brand tone is scored by held-out human raters on a sampled slice. LLM-as-judge is used only for the structured dimensions (resolution, tool accuracy) where the rubric is mechanical. The split matters: judging tone by another LLM produces grade-inflation that benefits exactly the vendors who optimize for it.

Why we run it on ourselves

Vorel runs the harness against itself the same way every other vendor does. Our scores get the same treatment as theirs. If we slip on a dimension, the published table shows it. We have considered, repeatedly, building a private 'Vorel-only' version that flatters our scores. We have refused every time, because the moment we do, the benchmark stops being credible (to us most of all).

The Stanford NLP partnership exists for the same reason. Academic partners review the rubric, the rater qualifications, and the scoring math. They publish their critique. We update the methodology. The benchmark gets better and Vorel does not get a vote that anyone else does not have.

What we are not claiming

CXBench is not a replacement for evaluating an AI vendor against your specific business. Your customers, your CRM, your specific use cases, and your specific failure modes are not on the benchmark (they cannot be). The benchmark is a portable, repeatable signal. The actual decision should still be made on a pilot in your environment.

CXBench is also not the only thing that matters. CSAT, NPS, and real-customer retention are the metrics a vendor's existing customers actually feel. They are not portable across deployments and they are easy to cherry-pick, which is why they do not belong on a benchmark. But they do belong in your evaluation.

What CXBench is, is a public, reproducible, defensible measurement of the things that are portable. v1 publishes Q3 2026 at cxbench.ai/results. We are excited to see how the leaderboard looks, including the line for Vorel.