Voice agent builders at Boardy and Flagler Health hit the same production walls

Two agents, one problem list

The panel featured Owen Gretzinger, Head of Engineering at Boardy, and Dylan Wight, a Software Engineer at Flagler Health. AssemblyAI's Ryan Seams moderated. Boardy makes introductions between founders, investors, customers and talent across a network of roughly 200,000 people. Flagler Health is an AI platform for musculoskeletal care. Its voice agents answer inbound scheduling calls and place outbound pre-procedure calls, telling patients which medications to stop, when to stop eating and what to expect on the day.

According to AssemblyAI, both teams struggle with trust, turn-taking, interruptions, latency, evals and memory. They also share infrastructure: LiveKit for the real-time layer, with Boardy using AssemblyAI for streaming speech-to-text and Flagler running a full AssemblyAI transcription pass on every call.

Before the panel, Seams demoed Universal-3.5 Pro Realtime. The demo included agent context, where the agent's own question is passed into the transcription request so short or mumbled replies resolve correctly. AssemblyAI says that across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2%.

Open conversation versus a flowchart

The sharpest disagreement was about how much freedom a call should have. Boardy leans into open-ended talk, because the more someone speaks, the more it learns about their goals and the better its intros get. "When you talk with someone over voice, you just say so much more," Gretzinger said. "Whereas over text you're very dry."

Flagler does close to the opposite. Its calls follow a flowchart, such as collecting an email, a phone number or insurance information, or delivering a specific set of pre-op instructions. "It's a big issue if it starts trying to give them medical advice," Wight said. When a patient goes off script, the call escalates to a human.

Success is defined differently too. Boardy tracks whether intros convert, and recently more than 50% of proposed intros turned into a real meeting. Flagler's goal is to avoid making the patient unhappy. "We're not really trying to delight this user," Wight said. The takeaway from AssemblyAI is that the right conversational behavior depends on who is calling and why, so a demo from another team is a poor template.

Disclose the AI in the first sentence

Flagler debated whether to tell patients they are speaking to an AI and decided to do so immediately on every call: "Hey, I'm Sarah, an AI assistant from this clinic, calling about your upcoming procedure." The reasoning was practical. In a handful of early calls, patients asked near the end whether they were talking to a person, and that moment cost more trust than disclosure would have.

Boardy also discloses, with more flair. Its opening line is often a question about whether this is the caller's first phone call with an AI. "That breaks the fourth wall almost immediately," Gretzinger said. Both teams concluded that it is better to disclose up front and spend effort on making the following minutes good.

Trust also has a technical side. Flagler's calls go out to numbers patients do not recognize, so the team spends real effort on Twilio configuration, caller ID that shows the clinic's name, and avoiding spam flags.

Turn-taking, latency and what the user actually heard

Gretzinger said one-on-one turn-taking is basically solved for Boardy with LiveKit and AssemblyAI. The harder problem is multi-party calls. Boardy now joins Google Meet calls, where it must decide whether anyone is talking to it. "False negatives and false positives are both catastrophic," Gretzinger said. Boardy uses a classifier that decides whether to speak based on the conversation so far, alongside a pipeline that generates responses in advance. Gretzinger said Boardy holds itself to a sub-second latency bar, since below about a second an agent can take part in the conversation.

Flagler's pre-op calls exposed another gap. When a patient interrupts partway through a block of instructions, the model knows what it meant to say but not how much the patient heard. "The agent as we have it right now doesn't listen to itself speak," Wight said. Flagler breaks instructions into smaller chunks and restarts the whole block after an interruption. Both are workarounds with tradeoffs: restarting is repetitive, and skipping ahead risks a missed instruction.

What it means for teams building voice agents

The panel's practical lesson is that production voice agents depend on context. A wandering conversation is the product at Boardy and a bug at Flagler. For anyone delivering information a person must act on, the gap between what an agent said and what a user heard needs explicit design. The telephony layer matters as well. Wight said more than half of Flagler's outbound calls go to voicemail, one of the surprises in moving from demo to production.