Perceived-Phone Recognition for L2 Speech

Automatic speech recognition is optimized to recover the words a speaker intended to say. It uses language models and lexicons to map speech to the most likely transcript.

When a learner mispronounces a sound, an ASR system can still return the expected word. A learner may say [sɹiː] and the transcript may still be “three.” That is useful for transcription, but it hides the pronunciation error.

Perceived-phone recognition answers a different question: what did a listener actually hear the speaker say? The goal is to identify each phone as the learner realized it, regardless of the word they intended to say.

Data is the bottleneck. Building this dataset requires trained linguists to listen to every recording and transcribe the phones they actually hear. The difficult part is separating the acoustic realization from the word the speaker was expected to say. Linguists must ignore lexical context and resist correcting an unusual pronunciation into a likely word. Ambiguous phones require repeated listening and sometimes adjudication. This makes annotation slow, expensive and difficult to scale.

Among widely used English corpora, manually verified phone labels are essentially limited to L2-ARCTIC and TIMIT. L2-ARCTIC provides roughly 3.6 hours of manually reviewed L2 read speech and 26.1 minutes of annotated spontaneous speech. TIMIT adds about five hours of hand-verified phone transcripts, but all of its speakers are native American English speakers. This leaves about four hours of relevant L2 data, almost all of it read speech, with no commercially reusable L2 corpus in this comparison.

Phone error rate (PER) counts substitutions, insertions and deletions against one primary human reference. Phone feature error rate (PFER) reports normalized articulatory-feature error on the same broad-phone comparison. Lower is better for both metrics.

The three comparisons answer three different questions: how released recognizers transfer to L2 speech, what the FLUENTLY method contributes under public L2 supervision, and what changes when every model is adapted to real learner conversations.

The method matters. With the same L2-ARCTIC supervision, FLUENTLY ranks first at 13.34% PER. With the same FLUENTLY supervision, the strongest external model makes 23% more phone errors.

The data matters. Fine-tuning on FLUENTLY improves each non-diagnostic external model on real learner calls, reducing PER by 14–20% versus its released version.

The advantages compound. The full FLUENTLY pipeline reaches 20.97% PER versus 29.93% for the strongest released baseline: 30% fewer phone errors on held-out learner conversations.

Metrics. PER is measured against one strict broad-phone human reference; substitutions, insertions and deletions count as errors. PFER reports normalized broad-phone articulatory-feature error on the same output. Lower is better.

Evaluation. Every reported state is tested on the same 900 held-out L2-ARCTIC utterances and fixed 846-utterance FLUENTLY cohort. Training data, development data and both test sets are disjoint.

Projection. Every model uses the same deterministic broad-phone projector before scoring. Frozen systems retain their released decoder; tuned systems use a shared atomic broad inventory. Unknown symbols count as explicit errors rather than being dropped.

Selection. Fine-tuning configurations and checkpoints are selected only from training and development results. Test data does not influence inventories, projection, hyperparameters or checkpoint choice.

Scoring protocols. Under the shared benchmark protocol, FLUENTLY fine-tuned on L2-ARCTIC scores 13.34%. Under the publication-matched TIMIT-39 reduction, FLUENTLY scores 13.42% versus 14.13% reported by El Kheir et al., SLaTE 2023. The 13.34% and 14.13% values should not be compared directly.