Sierra's τ-i18n tests voice agents in five more languages, and Korean lags most

Same tasks, five more languages

The new benchmark grew out of an earlier finding. In its τ-Voice experiments, Sierra says, accents caused the largest average drop in task completion among the factors it tested. That led to a broader question: how well do voice agents serve callers in other languages?

τ-i18n, pronounced tau-internationalization, uses the same airline, retail and telecom tasks across languages. Tasks, policies, tools and success criteria stay fixed, which lets Sierra compare performance from one language to the next.

The benchmark is built around language packs that let native speakers define choices for each locale. According to Sierra, the packs adapt caller identities and matching database records, personas, voices, speech conventions and evaluation rules, while the underlying service tasks stay the same.

Localization beyond translation

Sierra distinguishes between internationalization, the groundwork that lets a product adapt to different languages, and localization, the work of fitting a particular one. In customer service that can mean names, dates, currencies, product terms and local conventions. Voice adds pronunciation, code-switching, forms of address and turn-taking.

The company gives concrete examples. In Hindi, an agent with a female voice might refer to itself in the masculine. In Mandarin, an agent might say a stiff phrase meaning "execute a cancellation operation" where a person would say "help you cancel". Both get the point across, Sierra says, but neither sounds quite right.

Name order is another case. Korean and Mandarin callers typically give their family name first, so an agent that puts the first spoken name into a given-name field can fail authentication even when it heard the name correctly.

Two new measures and what the results show

Task completion shows whether the agent reached the caller's goal, but not how the call went. τ-i18n therefore adds two measures, reported separately:

• Interaction: whether the agent responds or yields at the right time, avoids interruptions and long monologues, and ignores backchannels or stray sounds that need no reply (Sierra calls this selectivity). It also covers tool-use efficiency.
• Generation quality: what the caller actually hears, including naturalness (translationese, grammatical-gender mistakes, register, honorifics) and speech fidelity for words, names and numbers.

The results cover three voice agents and 2,700 primary calls. On task completion, Spanish, Hindi and Portuguese trail English by 2.4, 3.8 and 6.4 percentage points on average. Korean and Mandarin trail by 22.6 and 9.1 points. Korean and Mandarin calls also have the most tool failures for every agent.

Responsiveness is a common weakness. According to Sierra, GPT Live 1 and Gemini 3.8 Live miss a caller turn, reply slowly or fail to yield in more than 80% of calls, against 16% for Grok Voice Think Fast 2.0. In 26 to 60% of calls, agents respond more than three times to backchannels, vocal tics or speech not directed at them.

GPT Live 1 leads task completion at 79.9% and generation quality at 63.4%. Grok Voice Think Fast 2.0 leads interaction at 52.6% but scores 17.7% on generation quality. In Sierra's words, "the most responsive agent is not necessarily the most natural speaker."

Open to more languages

Sierra calls the five languages "a starting point". It has open sourced its codebase and LanguageFactory, the framework behind the language packs, so native speakers, researchers and language communities can add languages and locales. Contributors will be able to build representative callers, review voices and sample conversations, and define the conventions agents should be evaluated on.

For teams deploying voice agents outside English, the results suggest that a single headline score can hide large gaps by language, and that finishing a task does not mean the call sounded right or flowed well. The work is credited to researchers at Sierra, Princeton University and Mercor.