Perceived-Phone Recognition for Accented English

Speech recognition is built to recover the words a speaker meant to say. Its language models and lexicons map speech to the most likely transcript, so when a learner mispronounces a sound it can still return the expected word. A learner may say [sɹiː] instead of [θɹiː] and the transcript still reads “three”. That serves transcription but hides the pronunciation error.

Perceived-phone recognition asks what a listener actually heard. It identifies each phone as the learner realized it, whatever word they meant to say (Table 1).

This benchmark compares five phone recognition model families on two held-out test sets, read accented speech from L2-ARCTIC and real learner conversations from FLUENTLY-L2, our own dataset. It reports two metrics. Phone error rate (PER) counts substituted, inserted and deleted phones as a percentage of the phones in the reference. Phone feature error rate (PFER) works like PER but counts errors in articulatory features such as voicing, place and manner of articulation. A near miss such as [s] for [θ] therefore costs less than an unrelated phone. Both metrics are reported for released models, for every model fine-tuned on L2-ARCTIC and for every model fine-tuned on FLUENTLY-L2.

Labeled phone data for accented speech is scarce. Trained linguists have to listen to every recording and transcribe the phones they actually hear, not the word the speaker was expected to say. They must ignore lexical context and resist correcting an unusual pronunciation into a likely word. Ambiguous phones need repeated listening and sometimes adjudication. Annotation is therefore slow, expensive and hard to scale.

Among widely used English corpora, manually verified phone labels are essentially limited to L2-ARCTIC and TIMIT (Table 2). TIMIT has no non-native speakers, which leaves about four hours of accented speech, almost all of it read and none of it licensed for commercial use.

Our dataset. Fluently is an AI language tutor used by non-native English speakers from more than 160 countries. We selected speech across 16 accents. Our in-house team of over 30 trained phonologists labeled the phones they heard. The resulting FLUENTLY-L2 dataset holds 3.8 hours of labeled training speech and a held-out 0.6-hour test set, recorded in the noisy, conversational conditions our product works in. It supports training, fine-tuning and benchmarking.

Test sets. L2-ARCTIC measures controlled read speech and FLUENTLY-L2 real learner–tutor conversations (Table 3).

Models. We compare Fluently PR with four released phone recognizers: ZIPA, XLSR-53, POWSM-CTC and MauBERT-phone. Fluently PR is our proprietary speech representation model. It represents how speech was produced, not only what was said. The representation can be paired with different pretrained speech encoders and adapted to downstream phone-recognition tasks. Table A1 lists the setup of each model.

Metrics. PER counts substituted, inserted and deleted phones as a percentage of the phones in one strict broad-phone human reference. PFER is the normalized broad-phone articulatory-feature error on the same output. It counts mismatched articulatory features such as voicing, place and manner of articulation, so a near miss costs less than an unrelated phone. Lower is better for both.

Evaluation. Every model is tested on the same clips of both test sets. Training data, development data and both test sets are disjoint.

Projection. Every model goes through the same deterministic broad-phone projector before scoring. Released models keep their own decoder and fine-tuned models share one atomic broad inventory. Unknown symbols count as errors rather than being dropped.

Selection. Fine-tuning configurations and checkpoints are selected on training and development results only. Test data does not influence inventories, projection, hyperparameters or checkpoint choice.

This report is a preview of FLUENTLY-BENCH-01, released in August 2026. To suggest a model or discuss the benchmark, write to [email protected].

To cite this report: