Apple researchers detect code-switching directly from speech encoders
Apple researchers have proposed a lightweight method that detects code-switching, the alternation between languages within one utterance, directly from speech audio. Using the Whisper-small encoder, the approach reached a mean F1 of 0.798 across 14 language pairs in CS-FLEURS, according to the paper.
- Apple researchers detect code-switching directly from speech, without transcription.
- The method finds language-selective hidden-state dimensions using monolingual recordings.
- Whisper-small reached a mean F1 of 0.798 across 14 language pairs in CS-FLEURS.
- Baselines scored 0.776 (CNN) and 0.751 (HuBERT linear).
- The authors say it can help build more representative multilingual corpora.
Why code-switching data is hard to find
Code-switching (CS) is common in multilingual speech, but it is underrepresented in pretraining corpora, the Apple team writes. Making speech models better at handling it requires more CS examples in training data, and that in turn requires finding the relevant utterances inside large speech collections.
One existing route is to transcribe the speech and then look for language alternation in the text. According to the researchers, this depends on obtaining accurate multilingual transcriptions, which is a limiting factor. Their paper, titled "Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models", skips the transcription step.
How the method works
The approach uses frame-level language information already present in pretrained speech encoders to decide whether both languages appear within an utterance. The authors first identify language-selective hidden-state dimensions using monolingual recordings. They then aggregate the frame-level activations of those dimensions into a CS prediction.
A small CS set is used to tune a decision rule. The authors describe the overall method as lightweight, since it builds on encoders that are already pretrained rather than on a new transcription pipeline.
- Identify language-selective dimensions from monolingual recordings.
- Aggregate frame-level activations of those dimensions.
- Tune a decision rule with a small CS set to predict code-switching.
Results on CS-FLEURS
The researchers report that the Whisper-small encoder achieves a mean F1 of 0.798 across 14 language pairs in CS-FLEURS. For comparison, a CNN reached 0.776, and a HuBERT linear baseline trained for monolingual language identification reached 0.751.
The margins are modest, but the method relies on a small tuning set and on information already inside the encoder. The authors conclude that pretrained speech encoders already encode the information needed to identify CS utterances.
What it means
According to the paper, the method offers a lightweight tool for building more representative multilingual corpora. If teams can flag code-switched utterances directly in raw audio, they can add more of them to training sets without first producing multilingual transcripts.
The work was authored by Jie Chi, Zakaria Aldeneh, Masha Fedzechkina, Maureen de Seyssel and Natalie Schluter, and was published in October 2026. It sits alongside other Apple research on code-switching, including an earlier paper on iterative pseudo-labeling for Mandarin-English code-switching speech recognition.
