Scale AI launches Voice Showdown, the first real-world benchmark for voice AI — and the results are humbling for some top models

Voice AI is moving faster than the tools we use to measure it. Every major AI lab — OpenAI, Google DeepMind, Anthropic, xAI — is racing to ship voice models capable of natural, real-time conversation.

But the benchmarks used to evaluate those models are largely still running on synthetic speech, English-only prompts, and scripted test sets that bear little resemblance to how people actually talk.

Scale AI, the large data annotation startup whose founder was poached by Meta last year to lead its Superintelligence Lab, is still going strong and tackling the problem head on: today it launches Voice Showdown, what it calls the first global preference-based arena designed to benchmark voice AI through the lens of real human interaction.

This product offers a unique strategic value to users: free access to the world’s leading frontier models. Through Scale’s ChatLab platform, users can interact with high-tier models—which typically require multiple $20-per-month subscriptions—at no cost. In exchange, users participate in occasional blind, head-to-head "battles" to choose which of two anonymized leading voice models offers a better experience, providing data for the industry’s most authentic, human-preference leaderboard of voice AI models.

"Voice AI is really the fastest moving frontier in AI right now," said Janie Gu, product manager for Showdown at Scale AI. "But the way that we evaluate voice models hasn't kept up."

The results, drawn from thousands of spontaneous voice conversations across more than 60 languages, reveal capability gaps that other benchmarks have consistently missed.

How Scale's Voice Showdown works

Voice Showdown is built on ChatLab, Scale's model-agnostic chat platform where users can freely interact with whichever frontier AI model they choose — for free — within a single app. The platform has been available to Scale's global community of over 500,000 annotators, with roughly 300,000 having submitted at least one prompt. Scale is opening the platform to a public waitlist today.

The evaluation mechanism is elegant in its simplicity: while a user is having a natural voice conversation with a model, the system occasionally — on fewer than 5% of all voice prompts — surfaces a blind side-by-side comparison. The same prompt is sent to a second, anonymous model, and the user picks which response they prefer.

This design solves three problems that plague existing voice benchmarks.

First, every prompt comes from real human speech — with accents, background noise, half-finished sentences, and conversational filler — rather than synthesized audio generated from text.

Second, the platform spans more than 60 languages across 6 continents, with over a third of battles occurring in non-English languages including Spanish, Arabic, Japanese, Portuguese, Hindi, and French.

Third, because battles occur within users' actual daily conversations, 81% of prompts are conversational or open-ended — questions without a single correct answer. That rules out automated scoring and makes human preference the only credible signal.

Voice Showdown currently runs two evaluation modes: Dictate (users speak, models respond with text) and Speech-to-Speech, or S2S (Speech-to-Speech, users speak, models talk back). A third mode — Full Duplex, which captures real-time, interruptible conversation — is in development.

Incentive-aligned voting

One design detail sets Voice Showdown apart from Chatbot Arena (LM Arena), the text benchmark it most closely resembles. In LM Arena, critics have noted that users sometimes cast throwaway votes with little stake in the outcome. Voice Showdown addresses this directly: after a user votes for the model they preferred, the app switches them to that model for the rest of their conversation. If you voted for GPT-4o Audio over Gemini, you're now talking to GPT-4o Audio. That alignment of consequence with preference discourages casual or dishonest voting.

The system also controls for confounds that could corrupt comparisons: both model responses begin streaming simultaneously (eliminating speed bias), voice gender is matched across both options (eliminating gender preference bias), and neither model is identified by name during voting.

The new Voice AI leaderboard every enterprise decision-maker should pay attention to

Voice Showdown launches with 11 frontier models evaluated across 52 model-voice pairs as of March 18, 2026. Not all models support both evaluation modes — the Dictate leaderboard includes 8 models, while S2S includes 6.

Dictate Leaderboard (Speech-In, Text-Out)

In this mode, users provide a spoken prompt and evaluate two side-by-side text responses. Here are the baseline scores:

Gemini 3 Pro(1073)Gemini 3 Flash(1068)GPT-4o Audio(1019)Qwen 3 Omni(1000)Voxtral Small(925)Gemma 3n(918)GPT Realtime(875)Phi-4 Multimodal(729)

Note: Gemini 3 Pro and Gemini 3 Flash are statistically tied for the top rank.

Speech-to-Speech (S2S) Leaderboard

In this mode, users speak to the model and evaluate two competing audio responses. Also baselines:

Gemini 2.5 Flash Audio(1060)GPT-4o Audio(1059)Grok Voice(1024)Qwen 3 Omni(1000)GPT Realtime(962)GPT Realtime 1.5(920)

Note: Gemini 2.5 Flash Audio and GPT-4o Audio are statistically tied for the top rank in baseline evaluations.

Dictate rankings are led by Google's Gemini 3 Pro and Gemini 3 Flash, which are statistically tied at #1 with Elo scores around 1,043-1,044 after style controls.

GPT-4o Audio holds a clear third place. Open-weight models including Gemma3n, Voxtral Small, and Phi-4 Multimodal trail significantly.

Speech-to-Speech (S2S) rankings show a tighter race at the top, with Gemini 2.5 Flash Audio and GPT-4o Audio statistically tied at #1 in the baseline rankings.

After adjusting for response length and formatting — factors that can inflate perceived quality — GPT-4o Audio pulls ahead (1,102 Elo vs. 1,075 for Gemini 2.5 Flash Audio).

Grok Voice jumps to a close second at 1,093 under style controls, suggesting its raw #3 ranking undersells its actual performance quality.

Qwen 3 Omni, the open-weight model from Alibaba's Qwen team, performs better on pure preference than its popularity would suggest — ranking fourth in both modes, ahead of several higher-profile names.

"When people come in, they go for the big names," Gu noted. "But for preference, lesser-known models like Qwen actually pull ahead."

Surprised revealed by real-world preference data

Beyond rankings, Voice Showdown's real value is in the failure diagnostics — and those paint a more complicated picture of voice AI than most leaderboards reveal.

The multilingual gap is worse than you think

Language robustness is the starkest differentiator across models. In Dictate, Gemini 3 models lead across essentially every language tested.

In S2S, the winner depends heavily on which language is being spoken: GPT-4o Audio leads in Arabic and Turkish; Gemini 2.5 Flash Audio is strongest in French; Grok Voice is competitive in Japanese and Portuguese.

But the more alarming finding is how frequently some models simply stop responding in the user's language at all.

GPT Realtime 1.5 — OpenAI's newer real-time voice model — responds in English to non-English prompts roughly 20% of the time, even on high-resource, officially supported languages like Hindi, Spanish, and Turkish.

Its predecessor, GPT Realtime, mismatches at about half that rate (~10%). Gemini 2.5 Flash Audio and GPT-4o Audio sit at ~7%.

The phenomenon runs both directions: some models carry non-English context from earlier in a conversation into an English turn, or simply mishear a prompt and generate an unrelated response in the wrong language entirely.

User verbatims from the platform capture the frustration bluntly: "I said I have an interview today with Quest Management and instead of answering, it gave me information about 'Risk Management.'"