Test Arabic ASR Benchmark
8 speech-to-text models on kadirnar/test-arabic-dataset: 21 hard recordings — two-channel call-centre calls, Saudi, Emirati, Jordanian, Lebanese and Egyptian Arabic, Arabic–English code-switching, and English. Lower is better.
552speech segments
78.4 minaudio
11,819reference words
4,707 / 7,092Arabic / English words
Soniox leads overall, but only through English. On Arabic speech the run-5 models are better (15.3–16.3 WER against
Soniox's 17.9); on English Soniox is far ahead (5.5 against 7.5–8.3). Overall, the gap between Soniox and run 5 is +0.95 WER,
at the edge of the paired-bootstrap 95 % interval (−0.07 to +1.95). The run-4 / run-5 models, run5base (run 5's recipe trained from the base model) included, are tied with each other and
are about 4.5 points better than run 2 / run 3 and 7 points better than the base model.
Per recording
How it was measured
- References. The transcripts in the dataset's
metadata.jsonl, accepted by the dataset owner as ground truth. They were made from five commercial speech-to-text systems (ElevenLabs Scribe v2, Meta Muse Voice Transcribe 1.0, Microsoft MAI-Transcribe-2, OpenAI GPT-Transcribe, xAI Grok STT 1.0), combined by a ROVER vote (a word needs two systems) and reviewed segment by segment, in conversation order, by Claude (an LLM; it did not listen to the audio). Those five systems are not on the board, because the references were built from their outputs. - Conventions. Dialect as spoken; English in Latin script except loanwords the client writes in Arabic script (اوكي، واتساب، ايميل …); numbers as digits; no sound tags.
- Segments. Each recording is cut at pauses into segments of at most 30 s; two-channel calls one channel (speaker) at a time. 441 segments were marked certain in the review (Reliable); 61 hold no speech, so every word a model writes there is an insertion (Words on silence). Arabic / English follow the script of the reference.
- Scoring. Corpus-level error rates, (S + D + I) / reference words or characters, after the extended normaliser of smart-turn-arabic-asr-benchmark on both sides (number words → digits, ة/ه, hamza and alef forms, punctuation, fillers). 95 % confidence interval: 1,000 bootstrap resamples of segments; models whose intervals overlap are not clearly different.
- Decoding. Qwen3-ASR models: vLLM, bf16, greedy, the model's own language detection (the set mixes Arabic
and English), no system prompt (the fine-tunes' default: the client's written format), repetition repair.
Soniox: its async API (
stt-async-v5), one request per segment, language hintsar, en.