Audible Builds Controllable TTS to Give Audiobook Characters Distinct Voices

Why standard TTS struggles with audiobooks

Text-to-speech systems sound natural in chatbots and virtual assistants, where interactions last seconds. Longer content is harder. According to Amazon Science, conventional TTS models fail to reliably change speaking style between characters, as a human narrator would. That leaves listeners unable to tell narration from dialogue, or to know which character speaks which lines.

The post gives several reasons. Small flaws such as unnatural prosody or monotone delivery are tolerable for short stretches but become grating over time, and audiobook listeners spend 10 to 15 hours with the same voice per book. General-purpose models also process individual utterances with minimal context, while an audiobook depends on knowing the surrounding story, sometimes from earlier chapters. Finally, such models often voice a character in a markedly different way in chapter 1 than in chapter 10.

How the controllable TTS model is trained

Audible's approach lets the model be steered with natural-language descriptions, for example "speak in an enthusiastic, high-pitched voice". This allows the TTS model to draw on the textual understanding of large language models. Training such a model requires speech data annotated with how it sounds, and the data must be expressive across pitch, speaking rate, character voices, emotions and speaking styles. Most existing training data is neutral in tone, the team says.

The team expanded its training sets through new recordings and human captioning of existing audio. Because existing datasets describe prosody in inconsistent formats, they used LLMs to rewrite the original prompts in varied lengths and structures, from single tags to free-form language. Human data collection does not scale to the needed size, so the team added synthetic data through two methods:

• Expert-model pipeline: expert models and signal-processing algorithms extract features such as pitch, speaking rate and emotion from audio, and an LLM turns these into speech captions.
• Speech captioner: a model that does the opposite of controllable TTS, taking speech and outputting a description of its prosodic features. It annotates more data, which trains the initial controllable-TTS model and further trains the captioner itself, creating what the team calls a synthetic-data flywheel.

BookLLM and the full audiobook pipeline

For the audiobook setting, the team built BookLLM, a component that uses off-the-shelf LLMs to identify unique characters and define their voice characteristics from details such as age, gender, personality and role. It then determines who speaks where and captions dialogue with delivery instructions. It was evaluated on 25 English and Spanish books in the mystery, sci fi and romance genres, against ground truth annotations from two human annotators. Accuracy in identifying which character speaks where, including separating narration from dialogue, was 97% in English and 94% in Spanish.

Given a new book, BookLLM first reads the whole text to build a cast list, ranks characters by importance and assigns each a distinct profile from a curated voice catalogue. It deliberately maximizes contrast between the most prominent characters. Minor and one-off characters fall back to the default narrator to avoid a proliferation of barely used voices. For each line of dialogue it writes a caption such as "whispering nervously" or "shouting in anger".

At synthesis time, each unit of text goes to the TTS model with its speaker's voice profile and, if needed, its caption. Since a character's voice profile is fixed once at the book level, the model renders that character consistently from beginning to end, according to the post.

Results and what they suggest

Using the pipeline to synthesize audiobooks with distinct character voices improved dialogue differentiation by 3.7 points on a MUSHRA test, a standardized method where listeners rate audio quality on a scale from 0 to 100. Across tests in English and Spanish, the team reports statistically significant improvements in listening experience for eight out of ten voices.

The source describes research results rather than a product launch, and it does not say when or whether the system will reach Audible's catalogue. What it does show is a direction for synthetic narration: separating story understanding, handled by an LLM, from speech rendering, so that long-form audio can keep characters recognizable across a whole book.