BASIS-Bench tests whether speech LLMs personalize without stereotyping

October 29, 2026, 18:14Source: NAVER LABS Europe

Researchers at NAVER LABS Europe have introduced BASIS-Bench, a benchmark that separates helpful personalization from stereotypical behavior in speech large language models. In tests on eight recent speech LLMs, they found that most models often recognize demographic cues but fail to turn them into appropriate personalization.

Key points

NAVER LABS Europe researchers introduced BASIS-Bench, a benchmark for speech LLMs.
It separates beneficial personalization from stereotypical behavior across three axes: demographic robustness, stereotypical bias and personalization.
Eight recent speech LLMs were evaluated; most recognize demographic cues but fail to personalize appropriately.
Explicit demographic labels improve personalization but amplify stereotypes already present in the textual backbones.
The authors call for training models to learn when demographic cues should shape responses.

Why speech LLMs raise a new bias question

Recent speech LLMs can pick up on rich paralinguistic cues, the non-verbal information carried by a speaker's voice. According to the authors, this sensitivity allows more complex and personalized interactions. It also creates a risk: a model that hears who is speaking may start to stereotype them.

The paper, titled "Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?", argues that little work has examined this link. Existing evaluations, the authors say, largely measure demographic disparities without distinguishing beneficial personalization from stereotypical behavior. The study is by Kshitij Ambilduke, Caroline Brun, Thibaut Thonet, Vassilina Nikoulina and Marcely Zanon Boito, and was published on 29 October 2026 (arXiv:2610.13550v1).

Three axes of evaluation

BASIS-Bench is built to disentangle the two aspects through three axes:

Demographic robustness: measures disparities in content-based tasks across speakers, as well as the models' demographic classification capabilities.
Stereotypical bias: measures whether perceived demographic cues trigger unprompted stereotypical behavior.
Personalization: measures whether models can tailor responses to demographic characteristics when explicitly requested.

The split matters because a model can be right to adapt its answer when a user asks for that, yet wrong to adapt on its own based on assumptions about the speaker. A benchmark that only counts demographic differences cannot tell these cases apart.

What the eight models showed

The team evaluated eight recent speech LLMs. They report that, despite often recognizing demographic cues, most of the models fail to translate them into appropriate personalization.

The researchers also tested explicit demographic labels. These improve personalization, but they also amplify stereotypical associations already present in the textual backbones, the text-based language models that the speech systems build on. In other words, giving the model clearer information about the speaker helps it tailor responses, yet it also brings out biases the underlying text model already carried.

What it means for developers

The authors conclude that speech LLMs need to be trained not only to perceive demographic cues, but "to learn when they should shape their responses." For developers of voice assistants and other speech-driven systems, the benchmark offers a way to check both sides of the problem rather than a single bias score. The source does not name the eight models or give per-model results, so the details of how individual systems performed are in the full paper.