Apple Study: Teaching Speech Models to Tell Languages Apart Narrows Multilingual Gap
Apple researchers report that strengthening a speech model's ability to discriminate between languages during pretraining reduces, and on some measures closes, the performance gap between multilingual and monolingual self-supervised models. The paper, by Maureen de Seyssel, Jie Chi and Zakaria Aldeneh, tests the idea in a controlled English/French HuBERT setting.
- Apple researchers show that stronger language discrimination during pretraining narrows the multilingual gap in HuBERT models.
- Two interventions were tested: an auxiliary language classifier and per-language k-means targets.
- Phone-ABX error fell from 11.6% to 10.4%, against 10.8% for the monolingual model.
- sWUGGY rose from 52.1% to 56.7% (monolingual: 58.5%), and ProsAudit from 68.9% to 72.9% (monolingual: 72.6%).
- Gains were strongest when language discrimination was introduced in the first training iteration.
The multilingual gap
Self-supervised speech models can benefit from sharing information across languages. According to the paper, however, under a matched total pretraining data budget they still fall short of monolingual models. This shortfall is known as the multilingual gap.
The Apple team argues that making the model better at telling languages apart during pretraining helps reduce that gap, while still preserving substantial cross-language sharing. The work was published on Apple's Machine Learning Research site in October 2026 and falls under the Speech and Natural Language Processing research area.
Two interventions and the results
The researchers used a controlled English/French HuBERT setup and tested two ways of strengthening language discrimination: an auxiliary language classifier and per-language k-means targets.
Across these interventions, the authors report the following changes against the bilingual baseline:
- Phone discrimination (phone-ABX, lower is better): error fell from 11.6% to 10.4%. The monolingual model scored 10.8%.
- Lexical performance (sWUGGY, higher is better): rose from 52.1% to 56.7%. The monolingual model scored 58.5%.
- Prosodic performance (ProsAudit, lexical subtask, higher is better): rose from 68.9% to 72.9%. The monolingual model scored 72.6%.
On phone discrimination and the prosodic measure, the bilingual model with the interventions therefore matches or passes the monolingual figures reported in the paper. On the lexical measure, a gap remains.
Timing matters
The authors also looked at when the intervention is applied across HuBERT training stages. According to the paper, the strongest gains on most linguistic measures occur when language discrimination is introduced in the first iteration. Later or repeated interventions yielded smaller improvements and were accompanied by increased language-wise segregation, meaning the model's representations separated more strongly by language.
The researchers conclude that these results support a causal role for language discrimination in reducing the additional cost of multilingual learning.
Related work and what it means
The paper follows earlier Apple research on the same problem. A September 2025 ICASSP paper explored using audio-visual data to reduce the multilingual gap in self-supervised speech models, noting that multilingual models tend to underperform monolingual ones on each individual language, especially in bilingual settings. A June 2025 EMNLP paper introduced ABX-style tasks for evaluating how multilingual language models represent language identity and meaning.
For developers of multilingual speech systems, the findings suggest that explicit language discrimination early in pretraining is one route to narrowing the cost of covering several languages in one model. The evidence in the paper is limited to a controlled English/French HuBERT setting, so how well the approach carries over to more languages is not addressed in the source.
