Benchmarking Real-Time Speech Recognition on Accented English

The benchmark holds 5,381 clips of English spoken by 2,042 learners, 22.2 hours of audio with 118,218 words in the reference transcripts. Every clip is a learner’s side of a real voice conversation with Fluently’s AI tutor. The clips span 16 accent groups and 3 fluency levels, and each of the 48 combinations holds a similar amount of speech. This section describes how the audio was selected and how the reference transcripts were made.

The audio comes from Fluently’s mobile app on iOS and Android, from users who opted in to data sharing. In each session a learner talks with the AI tutor, and we kept only the learner’s own audio channel, so no clip contains the tutor’s voice. The speech is unscripted. Learners answer the tutor in their own words, with the pauses, repetitions, false starts and self-corrections of someone speaking a language they are still learning.

A clip is one whole learner turn of about 3 to 30 seconds, with a median of 13.5 seconds. Whole turns keep each answer intact, as a tutor receives it, and each clip’s end was extended to the natural end of speech, located with Silero VAD. Before selection, recordings were screened for personal or sensitive information and, with an automatic check per speaker, for children’s voices. Recordings that failed either check were removed.

The accent group comes from the learner’s native language and country, not from listening. Broad groups such as Arabic, East Slavic and Latin American span several countries (Table 1). The fluency level is the learner’s usual speaking rate in Fluently, not a CEFR level: low under 70, medium 70 to under 90, high 90+ words/min (Table 2). Each of the 48 accent and fluency cells holds 24 to 30 minutes of speech, balanced by gender and platform. No speaker contributes more than 60 seconds, so no single voice dominates a cell.

The references are verbatim. A tutor has to respond to what the learner actually said, so the guideline keeps grammar errors, repetitions, false starts, cut-off words and fillers, and never corrects the learner’s English. It writes numbers as words and marks unclear or foreign speech with tags.

Two annotation studios transcribed every clip independently. Our in-house team of over 30 trained phonologists resolved every disagreement span by span against the audio. Both studios followed this guideline, and clips where their transcripts agree after normalization keep the agreed text.

One studio worked from ElevenLabs Scribe v2 drafts, which favors Scribe. A blind re-transcription by our linguists measures the bias at 0.33 points for Scribe v2 Realtime (95% CI −0.11 to 0.77) and 1.47 points for Scribe v2 offline (95% CI 0.99–1.97). In real time the bias is within noise, so Scribe v2 Realtime and MAI-Transcribe-2 stay tied for first. The Scribe v2 rows carry *. The audio and the reference transcripts stay private, and this page publishes aggregate results only.

We evaluated 21 real-time systems from 16 providers, each through its streaming API. To measure what streaming costs, the same clips also went to 16 offline systems from 15 providers, 14 file APIs and 2 models that we ran ourselves. The Offline mode of the leaderboard shows them. For 11 real-time systems an offline system in the set runs the same model, so the gap between the two is the cost of streaming.

Every system received the audio alone, without conversation context. File APIs received the original recordings, 48 kHz mono, with two exceptions. Pulse Pro received 16 kHz, as its provider’s documentation asks, and Muse 24 kHz, the highest rate its API accepts. We set the language to English wherever the API accepts one and requested verbatim output wherever it offers that option, and otherwise kept each provider’s defaults. Table A1 lists every model with its settings and finalize signal.

The protocol is adapted from Artificial Analysis’s AA-WER Streaming. Each clip is streamed at real-time pace as 16-bit PCM in 100 ms chunks, at 16 kHz or at 24 kHz where the API requires it, so network delay is part of every measurement. Silero VAD marks the end of speech in each clip. Once the audio up to that point has been sent, the client sends the API’s finalize signal. Chirp 3 has no finalize signal, so we wait for its own final. The audio keeps flowing after the signal, first the rest of the clip and then silence, as it would from an open microphone.

We then wait at least 15 seconds for the first final transcript after the end of speech, and finals that arrive in a 5-second drain after the stream closes still count. The scored text is every final segment up to and including that first final. Several systems keep their provider’s default endpointing (Table A1), so 10 systems send some finals before our endpoint, and those count too. A stream that failed or ended without any final was streamed again, up to three attempts. An empty final that a model did return was kept and scored as an empty output.

Time to final runs from the moment the audio through the end of speech and the finalize signal have been sent to that first final, network time included, as in AA-WER Streaming. We report its mean, median (p50) and 95th percentile (p95). The median describes a typical turn and p95 the slow turn a user meets about once in twenty. We stop at p95 because the tail beyond it rests on few streams and depends mostly on provider load at the moment of the run.

Of the 21 systems, 15 streamed at once from a cloud machine in the central United States during one 3.5-hour window on October 13, 2026, with 8 to 48 concurrent sessions per provider, so they faced the same network and the same time of day. Another 5 streamed on October 16 from a second machine in the western United States, and Nari Qwen3-ASR 1.7B Fast on October 19 and 20 from a third machine in the United States. A control rerun of three earlier systems on 300 clips from the second machine matched their first runs within 0.1 WER points and 19 ms of median time to final. From the third machine it matched within 0.12 WER points, with median time to final 3 and 6 ms higher for Nova-3 and Scribe v2 Realtime and 47 ms lower for Universal-3.5 Pro. Latency from other regions and under other loads will differ. The appendix gives the full protocol and its four differences from the Artificial Analysis methodology.

Reference and system output pass through the same English normalizer, the one of the Hugging Face Open ASR Leaderboard at a pinned commit (Table A2). It lowercases, drops punctuation, bracketed tags and fillers, expands contractions, writes numbers as digits and uses American spelling. A system is therefore not penalized for formatting, for writing “25” where the reference says “twenty five”, or for leaving out “um”. Repetitions and false starts stay in both texts, so a system that cleans them up is charged with deletions.

The two texts are then aligned word by word with kaldialign 0.12, where a compound split in one text matches it joined in the other. WER is pooled: substitutions, deletions and insertions summed over all clips of a slice, divided by the reference words of those clips. We never average per-clip rates, which would let one error in a short clip outweigh several in a long turn.

A provider error, timeout or empty transcript counts as an empty output, so every word of its reference is a deletion. Every system is thus measured on the same 5,381 clips and 118,218 reference words.

We report WER with each clip’s errors capped at its reference length, so a runaway output counts as fully wrong but cannot outweigh other clips. Offline systems produced 9 outputs with runaway repetitions, and real-time systems produced none. They start on ordinary disfluent speech, such as a repeated or stuttered word, and the other systems transcribe those clips normally. The leaderboard shows the plain pooled WER on hover.

Systems are ranked by WER, real-time and offline systems separately, and exact ties share a rank, as in 1, 2, 2, 4. Gaps of about 0.2 points are within sampling noise (paired speaker bootstrap). Scribe v2 Realtime and MAI-Transcribe-2 are tied for first, and Ink-2, Muse, Grok and Gemini 3.5 Live (ranks 4 to 7) are each tied with their neighbors.

Because every accent and fluency cell holds a similar amount of speech, no accent group dominates the overall WER, and neither does the mix of Fluently’s users. Pooling weighs every reference word equally, though, and slower speech holds fewer words per minute. Low-fluency learners supply 24% of the reference words and high-fluency learners 43%, although each level holds about a third of the audio.

The accent filter re-pools the data rather than averaging it. Selecting accent groups sums the errors and reference words of those groups alone, recomputes each system’s WER and ranks the systems again. The Low fluency by accent view does the same for low-fluency learners only. A system that ranks high overall can fall behind on a particular group, and the filter shows where. Accent filters shrink the sample, so ranks move more under them and small moves are noise.