Spotify Trains Conversational Agent on Synthetic Dialogs, Reports 14% Listening Gain

The cold-start problem: no real conversations before launch

Spotify Research published a post describing how it built a conversational recommendation agent, which lets users ask for music in natural language and then follow up, refine or change their requests. The challenge, according to Spotify, is that before launch there is no real multi-turn conversational data. Without users, the team cannot see how conversations unfold or where the agent fails.

Single-turn testing is not enough, Spotify argues, because it misses failures that only appear over a conversation. An agent might return a good playlist for "rock vibes" but fail when asked to narrow it down to niche 90s rock from the user's hometown, or forget an earlier constraint such as "only songs in Spanish."

Five capabilities and a two-stage generation pipeline

Spotify defined five core multi-turn capabilities, each with an evaluation rubric:

• Content Refinement: adapting when the user changes or refines a request.
• Instruction Retention: remembering constraints introduced earlier.
• Anaphora Resolution: resolving references to items from earlier turns, such as "who is the third artist?"
• Intent Shift: recognizing when the user switches to a new request.
• Separate Input: handling a request whose details arrive across several turns.

Conversations are generated in two stages. A dialog plan LLM takes a real single-turn seed prompt from existing experiences and turns it into an abstract plan aimed at one of the capabilities. A UserLLM then follows the plan while interacting with the live recommendation agent, so the result reflects real agent behavior, including tool calls and playlist outputs. Spotify says separating planning from execution lets plans be audited, versioned and replayed against different agent versions.

The company validated the output with human annotation. Across roughly 100 conversations, each rated by five annotators, 98% exercised the intended capability, 93% were judged realistic, 94% showed reasonable user adaptations and 97% had accurate judge verdicts.

What the evaluation revealed

Performance varied by capability. Intent Shift performed best, followed by Anaphora Resolution. Content Refinement and Instruction Retention were harder, especially when earlier constraints had to be kept across several turns. Quality also fell as conversations grew longer, with the largest degradation beyond four turns. Spotify says this motivates approaches such as context compaction.

The evaluation also pointed to concrete failures. In Anaphora Resolution, references to playlist positions sometimes failed because incorrect tool-call arguments changed track ordering. In Instruction Retention, the agent sometimes over-summarized a request when creating a music session and dropped earlier constraints.

A self-improvement loop for the agent prompt

Spotify then automated much of the manual prompt-fixing cycle. The loop samples multiple agent plans per query at elevated temperature and scores them with an LLM judge. If some plans succeed and others fail, Spotify treats it as a reliability failure. If all fail, it is a capability gap.

• Reliability failures use contrastive optimization: a coding agent compares successful and unsuccessful plans and proposes a prompt update that reinforces the successful behavior.
• Capability gaps use agentic iterative refinement: a coding agent analyzes failing traces and proposes changes to the prompt or tool configuration, such as a tool-use example or routing instruction.

The agent is re-evaluated after each update, and proposed changes are batched and reviewed by humans before merging. In one run, the loop improved pass@1 by 8% relative on the full test set, on top of an already highly optimized manual prompt. Fixes included making "give me more artists" refine an existing playlist, reading "hardcore" as "hardcore punk" in rock contexts, and using the full calendar year of listening history for "my year in music."

Online results and next steps

Spotify ran a two-week A/B test across several markets, reaching approximately 15 million users on multiple device types. The new experience, which supports multi-turn interactions, informational intents and music sessions, was compared with a baseline limited to music session refinement. According to Spotify, it increased additional user listening by 14% and weekly active users by 5%, while reducing skip rate by 5%. The agent was then rolled out to production.

Next, the company plans to extend its taxonomy to more conversational intents, bring in real production conversations as they become available, and integrate the self-improvement loop more broadly into the agent pipeline. The work is described in a paper published at RecSys 2026.

For teams building conversational agents, the account suggests that synthetic multi-turn tests and human-reviewed automated prompt fixes can stand in when real interaction data is scarce, though the reported figures come from Spotify's own evaluation.