CMU's MultiVerse Lets Artists Control How AI Adapts Their Lyrics

Adaptive lyrics with the artist in charge

MultiVerse was built by a research team that includes Alex Wang, a doctoral student in the Human Computer Interaction Institute (HCII) at CMU's Kenneth C. Griffin School of Computer Science. Wang is a musician and game developer, and the idea grew from his master's work in music. He said he was inspired by how dynamic video games are, since creators there design "a space of possibilities" in which each player has a unique experience within the creator's vision.

Adaptive lyrics means the person who made a song can specify which aspect of a lyric may be modified. The source gives an example: a song with a line about taking the train to New York City could be turned into versions where the destination changes to match where the listener is going, such as Pittsburgh.

How the steering controls work

MultiVerse takes a creator's lyrics and lets them adjust specific guardrails before the technology adapts the text. According to the researchers, there are three main categories of steering controls:

• Intent: feelings or themes.
• Structure: the rhythm or syllable count.
• Context: what audience information, such as location or mood, the lyrics adapt to.

Wang applied this to a song he wrote about returning to the Gates Hillman Center. Its opening lines are "Back in Gates, winding up the helix bend/Pausch lights flicker as the daylight ends." Elements such as the place, the routine and the sight can be turned into variable slots, for example "Back in [Familiar Place]".

The system generates multiple versions of the lyrics, and every version shown to the creator is computationally verified to meet the artist's constraints. If a lyric uses an ABAB rhyme scheme and MultiVerse produces an AABB scheme, that option is rejected.

Why the researchers say it matters

David Lindlbauer, an assistant professor in the HCII, pointed out that current generative AI can produce custom songs on almost any topic from a simple prompt. He said the results are likely to be nearly indistinguishable from human-written songs, but that "AI takes the artist and their creative process fully out of the loop, which is not necessarily desirable." The team sees music creation as a spectrum from fully manual to fully AI-generated and is exploring how to combine the benefits of both.

Chris Donahue, an assistant professor in the Computer Science Department, noted that music streaming services are increasingly partnering with artists and record labels to explore adaptive media in the form of AI song remixes. His question: "how can we build tools that let artists constrain the types of modifications these features can make to their music?"

Testing and next steps

The team tested the system with 10 songwriters. Their feedback showed that MultiVerse made them feel they had more control than they did with more hands-off generative AI tools. CMU's summary says the tool gave creators significantly greater control and confidence than general AI prompting.

The researchers will present their work at the Association for Computing Machinery's Symposium on User Interface Software and Technology (UIST) in Detroit. The source does not mention a public release or availability of MultiVerse.

For artists, the work points to a way of using AI personalization without handing over the song itself: creators decide what can change, and anything outside those limits is filtered out before they see it. For streaming services exploring AI remixes, it offers a model for building such features with artist-defined boundaries.