Twilio Adds GPT-Live Support to Agent Connect for Phone Voice Agents

What GPT-Live and Agent Connect do

According to Twilio, OpenAI's GPT-Live is a family of full-duplex speech-to-speech (S2S) models. The models separate the conversation from any necessary background work, which lets developers delegate that work to their own backend or to an OpenAI hosted model.

Twilio Agent Connect is described as Twilio's open-source TypeScript SDK for building voice and messaging AI agents. The new GPTLiveProvider works through a declarative config interface, and Twilio says developers do not need custom WebSocket plumbing. The tutorial opens with a familiar complaint about voice AI: it works in a testing script but falls apart on a real phone call.

The central class is VoiceChannel, which Twilio describes as provider-agnostic. The same class hosts Conversation Relay, GPT Realtime and GPT-Live, depending on the config passed to it.

How the tutorial works

The walkthrough uses TypeScript and builds an agent that greets the caller and answers a question that triggers a tool call. It then covers placing an outbound call with Twilio and GPT-Live, and mentions per-call customization. The full example is in the repository under getting_started/examples/gpt-live.

Prerequisites listed by Twilio:

• A Twilio account and a Twilio phone number with Voice capability.
• An OpenAI API key from a project with GPT-Live access.
• Node.js 22.13+ and the Twilio Agent Connect SDK v2.4.0+, installed with dotenv.
• A public HTTPS/WSS endpoint for local development, with ngrok suggested.

Setup starts with a TAC instance created from environment variables such as TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_API_KEY, TWILIO_API_SECRET, TWILIO_PHONE_NUMBER, TWILIO_VOICE_PUBLIC_DOMAIN and OPENAI_API_KEY.

The session is configured through defaultSessionConfig, which is sent as GPT-Live's session.start payload the moment the model connects. At minimum it needs the model, the instructions (the system prompt) and audio.format set to TWILIO_AUDIO_FORMAT_FOR_GPT_LIVE. Setting welcomeInstruction makes the agent speak first, which Twilio says is useful when callers do not know they have reached an AI agent.

Tool calls through delegation

According to the tutorial, GPT-Live does not call functions directly. It delegates tool calls either through Client Delegation, where the developer builds the delegation integration, or through a Responses API model. The tutorial uses Responses delegation.

A tool created with defineTool() is registered in tools so Twilio Agent Connect can execute it, and its schema is listed separately under delegation.responses.tools so the model knows it exists. The example is a get_weather function with a simulated 3-second delay.

Twilio says keeping tool execution outside the real-time voice loop means tool logic can be slow or complex, such as a database lookup, an API call or a business rule, while the call continues. GPT-Live hands the decision to call a tool to a Responses API model, then Twilio Agent Connect runs the TypeScript function and speaks the result back. Twilio suggests swapping the weather function for one that checks order status, appointment availability or account details.

Connecting the number

The server side takes a few lines: register the voice channel with tac.registerChannel, create a TACServer and start it. TACServer mounts the TwiML endpoint at /twiml by default.

In the Twilio Console, developers then open Phone Numbers > Manage > Active Numbers, select the number, and under Voice Configuration set "A call comes in" to Webhook with the URL https://<your-domain>/twiml and the HTTP method POST. Twilio says the result is a phone number that greets callers, answers questions and calls real functions, with no WebSocket code required.

What it means for developers

For teams building voice agents on Twilio, the provider offers a path to GPT-Live without writing the media streaming layer themselves. Because VoiceChannel is provider-agnostic, switching between Conversation Relay, GPT Realtime and GPT-Live is a matter of config rather than a rewrite, according to Twilio. Access to GPT-Live through an OpenAI project is still required.