LiveKit launches Agent Simulations to test voice agents before customers do

Why voice agents are hard to test

LiveKit argues that the happy path is the easy part of testing a voice agent, and often the only part that gets tested. Real callers may be on a bad connection, speak softly, give an account number in three pieces with a question in the middle, or try to make the agent say something it should not. The company says this long tail is where agents fail in production.

The language model at the core adds to the problem, because the same input will not always produce the same output. Testing by hand means calling your own agent repeatedly, which according to LiveKit does not scale with the number of tasks an agent performs or the languages its users speak.

The LiveKit Agents framework already supports turn-by-turn testing, but LiveKit says that approach cannot replicate the unpredictability of a real conversation. Agent Simulations instead works at the task level: you state the user's task, and the agent takes whatever path it needs to accomplish it.

How the simulations work

An LLM-driven user plays out each scenario from start to finish. For every scenario, developers can define the simulated user's personality, goals and behavior. Once the conversation reaches a natural conclusion, an LLM-based judge grades the transcript against freeform expectations set by the developer and assigns a pass/fail verdict with the reason behind it. Scenarios run in parallel, and results, transcripts and traces appear in the LiveKit Cloud dashboard.

Developers do not have to write scenarios from scratch. Running the CLI command lk agent simulate -n 10 makes LiveKit's scenario generation tool read the agent's code and produce scenarios based on its actual prompts, tools and conversation logic. LiveKit says this yields a pass rate on paths you never explored within minutes, without making a call yourself.

Generated scenarios are meant as a starting point. LiveKit suggests keeping those that surface problems, adjusting the rest, and adding custom cases. A checked-in scenarios.yaml file becomes the source of truth, versioned alongside the agent. Every scenario stays in the suite, so checks for earlier features run automatically when new ones are added. LiveKit notes that this matters because a voice agent has no interface to glance at, unlike a web app where a regression may be visible.

Text and audio modes

Text simulations skip the speech models and test only conversation logic. They are the default, fast and cost-effective at scale. LiveKit says they are scheduled at low priority and paced into spare capacity against the inference quota, so they do not compete with live traffic. They are cheap enough to run on every pull request, and a failed scenario makes the CLI return a non-zero exit code, which fails the build.

Audio simulations cover the entire audio pipeline, including background noise, interruptions and backchanneling, the sounds a listener makes while the other person is still talking. They work with cascaded and speech-to-speech/full duplex agents. They run in real time and are metered at a higher rate than text, so LiveKit recommends a nightly or pre-release schedule.

The simulated caller is designed to behave like a real one. It places small sounds where a listener would make them rather than on a timer. Near the end of a turn it uses only "mm-hmm" and "uh-huh," and drops "okay," so a backchannel does not accidentally answer the agent's question.

Availability and what it means for developers

According to LiveKit, Agent Simulations is free through October. Simulations run the real agent, so the developer's LLM is called as usual, and in audio mode so are STT and TTS, all billed at normal rates. Getting started requires the LiveKit CLI, LiveKit Observability enabled and a LiveKit Cloud project.

LiveKit says Serve AI used simulations to validate a full agent rewrite against a hundred real production calls before cutting any traffic over. For teams building voice agents, the pitch is that they can change agent code without wondering what they inadvertently broke, with tests that can run in CI on every pull request.