Vercel adds 14 ElevenLabs audio models to AI Gateway, at half price for now

Six types of audio models behind one key

According to Vercel, AI Gateway supports six types of ElevenLabs audio models, all reachable through a single gateway API key. The company describes them as follows:

• Speech to text: transcribe recordings or caption live audio as people speak.
• Streaming text to speech: start playing audio while a language model is still writing its reply.
• Realtime voice agents: let users talk to an app and hear it respond in a single live session.
• Music: describe a track in plain text and get a finished song or instrumental at the requested length.
• Sound effects: generate game and interface sounds, or ambience that loops cleanly, from a short description.
• Multi-voice dialogue: voice a whole conversation in one call, with a different voice for each speaker.

Vercel says an app or agent can transcribe what a user says, reason over it with a choice of hundreds of models, then answer in an ElevenLabs voice. Every request appears in the same usage, cost and trace views as other AI Gateway traffic.

The 14 model IDs include elevenlabs/scribe-v2 and elevenlabs/scribe-v2-realtime for transcription, four text to speech models such as elevenlabs/eleven-flash-v2.5, two realtime models, two music models, one sound effects model and three dialogue models.

Speech pipelines and realtime agents

Vercel shows two ways to build voice features. In the first, Scribe v2 Realtime converts speech to text while the user is still talking. A language model of the developer's choice writes the answer, and Eleven Flash v2.5, described as a low-latency voice model, reads it aloud as it streams in. Because each step is a separate model, the transcription, language or speech model can be swapped without changing the rest of the pipeline. Vercel's example has a user asking whether they will make a connecting flight.

The second approach uses Eleven Realtime v1 Max or Eleven Realtime v1 Mini, which handle the whole spoken conversation in one model over a single WebSocket session. Vercel says this removes the need for separate transcription and speech steps and suits support calls or in-app assistants.

For these sessions, the server only mints a short-lived token for the chosen model, so the API key stays off the client. In the browser, the useRealtime hook from @ai-sdk/react handles the connection, microphone capture and audio playback. The realtime models support semantic voice activity detection (VAD) to decide when a user has finished speaking, but they do not support server-vad.

Music, sound effects and dialogue

Music v2.5 and Music v2 compose original music from a text description. Users can name a genre, a mood or instruments, and the model returns a finished track at the requested length, with or without vocals. A song can also be laid out section by section, each with its own style and length. Sound Effects v2 works the same way for shorter sounds such as footsteps, a slamming door or looping rain.

All three models use generateSpeech, with text describing the desired sound. Vercel's example requests a 30-second track described as a mellow lo-fi beat with soft piano.

Eleven v4, Eleven v4 Turbo and Eleven v3 perform a script with several speakers in a single request. A voice is assigned to each line, and each change of voice starts a new turn. According to Vercel, a two-host podcast or a scene with a full cast comes back as one continuous recording, so no separate clips need stitching. Each voice is an ElevenLabs voice ID, and parts can stream in from a language model writing the script.

What it means for developers

For teams already using AI Gateway, the addition puts speech, music and sound generation alongside text models under one API, one set of usage and cost views, and the same controls. Vercel describes AI Gateway as offering retries, failover, budgets for API keys, routing rules, custom reporting and Zero Data Retention support.

On cost, Vercel says AI Gateway reflects provider pricing with no markup and charges no platform fee on inference, including on Bring Your Own Key (BYOK) requests. The 50% launch discount on ElevenLabs models runs until November 9th. Examples for all the models are in the AI Gateway audio docs.