Benchmark

Perceived-Phone Recognition for Accented English

How well five phone recognition model families hear the sounds non-native English speakers produce, in read speech and real conversations.

Hours
4.4
Accents
16
Test sets
2
Models
5
Contents
  1. 1Overview
  2. 2What’s in the benchmark
  3. 3Results
  4. 4Methodology
  5. 5Citation and contact
  6. 6References
  7. 7Appendix

1Overview

Speech recognition is built to recover the words a speaker meant to say. Its language models and lexicons map speech to the most likely transcript, so when a learner mispronounces a sound it can still return the expected word. A learner may say [sɹiː] instead of [θɹiː] and the transcript still reads “three”. That serves transcription but hides the pronunciation error.

Perceived-phone recognition asks what a listener actually heard. It identifies each phone as the learner realized it, whatever word they meant to say (Table 1).

StageExample
Intended word“three”
Expected phonesθɹiː
Heard phonessɹiː
Feedback“We heard [s] where [θ] was expected.”
Table 1. Expected and heard phones of “three”, aligned phone by phone. Only the first phone differs. Perceived-phone output keeps this substitution, which makes specific feedback possible.

This benchmark compares five phone recognition model families on two held-out test sets, read accented speech from L2-ARCTIC and real learner conversations from FLUENTLY-L2, our own dataset. It reports two metrics. Phone error rate (PER) counts substituted, inserted and deleted phones as a percentage of the phones in the reference. Phone feature error rate (PFER) works like PER but counts errors in articulatory features such as voicing, place and manner of articulation. A near miss such as [s] for [θ] therefore costs less than an unrelated phone. Both metrics are reported for released models, for every model fine-tuned on L2-ARCTIC and for every model fine-tuned on FLUENTLY-L2.

2What’s in the benchmark

Labeled phone data for accented speech is scarce. Trained linguists have to listen to every recording and transcribe the phones they actually hear, not the word the speaker was expected to say. They must ignore lexical context and resist correcting an unusual pronunciation into a likely word. Ambiguous phones need repeated listening and sometimes adjudication. Annotation is therefore slow, expensive and hard to scale.

Among widely used English corpora, manually verified phone labels are essentially limited to L2-ARCTIC and TIMIT (Table 2). TIMIT has no non-native speakers, which leaves about four hours of accented speech, almost all of it read and none of it licensed for commercial use.

CorpusClipsSpeakersHoursSpeechLicense
L2-ARCTIC3,59924≈3.6Read accented speech from six native languages, plus 26.1 min of spontaneous speechCC BY-NC 4.0, no commercial use
TIMIT6,300630≈5Read speech, native American English speakers onlyLDC, research only for non-members. Commercial use requires paid LDC membership.
Table 2. Public English corpora with manually verified phone labels. The L2-ARCTIC hours are estimated from its 3,599 manually examined clips at the published corpus-wide average of 3.6 seconds per clip.

Our dataset. Fluently is an AI language tutor used by non-native English speakers from more than 160 countries. We selected speech across 16 accents. Our in-house team of over 30 trained phonologists labeled the phones they heard. The resulting FLUENTLY-L2 dataset holds 3.8 hours of labeled training speech and a held-out 0.6-hour test set, recorded in the noisy, conversational conditions our product works in. It supports training, fine-tuning and benchmarking.

Test sets. L2-ARCTIC measures controlled read speech and FLUENTLY-L2 real learner–tutor conversations (Table 3).

Test setClipsHoursSpeechReferences
L2-ARCTIC900≈0.9Read speech from six held-out non-native speakersManually reviewed perceived phones
FLUENTLY-L28460.6Real learner–tutor conversations with naturally noisy audioHuman-labeled perceived phones
Table 3. The two held-out test sets. The L2-ARCTIC hours are estimated as in Table 2.

Models. We compare Fluently PR with four released phone recognizers: ZIPA, XLSR-53, POWSM-CTC and MauBERT-phone. Fluently PR is our proprietary speech representation model. It represents how speech was produced, not only what was said. The representation can be paired with different pretrained speech encoders and adapted to downstream phone-recognition tasks. Table A1 lists the setup of each model.

3Results

  1. Fluently PR leads under matched supervision. Fine-tuned on the same FLUENTLY-L2 data, Fluently PR scores 20.97% PER and 9.49% PFER on held-out learner conversations, against 25.69% and 12.38% for XLSR-53, the strongest of the other models. That is 18% fewer phone errors (Figure 1).
  2. Fluently PR also leads on read accented speech. Fine-tuned on the same L2-ARCTIC data, Fluently PR ranks first at 13.34% PER and 5.22% PFER, ahead of ZIPA at 14.67% and 6.34% (Table 5).
  3. Conversational data improves every external model. Fine-tuning on FLUENTLY-L2 lowers the PER of ZIPA, XLSR-53 and POWSM-CTC on learner conversations by 14–20% relative to their released versions (Tables 4 and 5). MauBERT-phone is left out because its released frame decoder is diagnostic.
  4. Released models transfer poorly to conversation. Every released model has a higher error rate on FLUENTLY-L2 conversations than on read L2-ARCTIC speech. The best released PER rises from 20.53% to 29.93% (Table 4).

PER on FLUENTLY-L2

  • Fluently PR
  • Other models
  1. Fluently PR, our model20.97%
  2. XLSR-5325.69%
  3. ZIPA26.31%
  4. POWSM-CTC27.81%
  5. MauBERT-phone28.98%
Figure 1. PER on held-out FLUENTLY-L2 learner conversations. All five models are fine-tuned on the same FLUENTLY-L2 training data. Lower is better.
ModelL2-ARCTIC PERL2-ARCTIC PFERFLUENTLY-L2 PERFLUENTLY-L2 PFER
ZIPA20.538.4131.2515.07
XLSR-5321.499.2429.9314.38
POWSM-CTC21.308.8434.8517.37
MauBERT-phone61.5830.9566.8236.54
Table 4. PER and PFER (%) of released models without task-specific fine-tuning. MauBERT-phone’s released frame decoder is diagnostic. Best in each column in bold. Lower is better.
ModelL2-ARCTIC PERL2-ARCTIC PFERFLUENTLY-L2 PERFLUENTLY-L2 PFER
Fluently PR13.345.2220.979.49
ZIPA14.676.3426.3114.59
XLSR-5317.347.2025.6912.38
POWSM-CTC16.546.8427.8113.53
MauBERT-phone18.087.8228.9812.93
Table 5. PER and PFER (%) of every model fine-tuned on L2-ARCTIC or FLUENTLY-L2 and tested on the held-out set of the same corpus. Best in each column in bold. Lower is better.

4Methodology

Metrics. PER counts substituted, inserted and deleted phones as a percentage of the phones in one strict broad-phone human reference. PFER is the normalized broad-phone articulatory-feature error on the same output. It counts mismatched articulatory features such as voicing, place and manner of articulation, so a near miss costs less than an unrelated phone. Lower is better for both.

Evaluation. Every model is tested on the same clips of both test sets. Training data, development data and both test sets are disjoint.

Projection. Every model goes through the same deterministic broad-phone projector before scoring. Released models keep their own decoder and fine-tuned models share one atomic broad inventory. Unknown symbols count as errors rather than being dropped.

Selection. Fine-tuning configurations and checkpoints are selected on training and development results only. Test data does not influence inventories, projection, hyperparameters or checkpoint choice.

5Citation and contact

This report is a preview of FLUENTLY-BENCH-01, released in August 2026. To suggest a model or discuss the benchmark, write to [email protected].

To cite this report:

BibTeX
@misc{fluently2026phone,
  title  = {Perceived-Phone Recognition for Accented English},
  author = {{Fluently}},
  year   = {2026},
  month  = aug,
  url    = {https://getfluently.app/research/phone-recognition}
}

6References

  1. [1]L2-ARCTIC documentation. PSI Lab, Texas A&M University.
  2. [2]TIMIT Acoustic-Phonetic Continuous Speech Corpus. Linguistic Data Consortium.
  3. [3]ZIPA. GitHub repository.
  4. [4]XLSR-53. Hugging Face model card.
  5. [5]POWSM-CTC. Hugging Face model card.
  6. [6]MauBERT-phone. Hugging Face model card.
  7. [7]Multi-View Multi-Task Representation Learning for Mispronunciation Detection. El Kheir et al., SLaTE 2023.

7Appendix

Model setups
ModelReleasedFine-tuned
Fluently PR—Proprietary speech representation
ZIPAZipformer, CR-CTC LargeFull-model fine-tune, broad CTC
XLSR-53Released phone decoderFull encoder fine-tune, broad CTC
POWSM-CTCMultilingual pronunciation-prompted CTCUpper-encoder fine-tune, broad CTC
MauBERT-phoneReleased frame decoder, diagnosticBroad CTC fine-tune
Table A1. Each model as released and as fine-tuned, with the same fine-tuned setup on L2-ARCTIC and FLUENTLY-L2. Fluently PR is not a released model.
Comparison with published results

Under the shared protocol of this benchmark, Fluently PR fine-tuned on L2-ARCTIC scores 13.34% PER. Under the TIMIT-39 reduction used by El Kheir et al. (SLaTE 2023), it scores 13.42%, against their reported 14.13%. The 13.34% and 14.13% values should not be compared directly.