Modal launches Endpoint Candidates to test inference changes on live traffic

What Endpoint Candidates do

According to Modal, latency, throughput and cost optimization are at the core of modern inference engineering, and none of these metrics can be reliably improved without testing and measurement. Endpoint Candidates are the first of several tools the company says it is building to help users optimize their own inference servers.

A Candidate is an alternative deployment attached to an existing Dedicated Endpoint. It can mirror production by handling a sample of production traffic, while Modal runs the underlying infrastructure. Teams can use it to test new model checkpoints, or even entirely different models, without building their own traffic-mirroring and deployment infrastructure, according to the company.

How to create, evaluate and promote a Candidate

Setup starts from an existing Dedicated Endpoint in a Modal environment. Users select Create candidate and choose either an open model from Modal's catalog or their own weights from a Hugging Face repo or a Modal Volume. Once the Candidate is ready, they configure a traffic sample to begin mirroring production requests to it.

Production keeps handling every original request and returns the response users see. The dashboard shows the Candidate's inference metrics, including latency and throughput, and supports performance benchmarks to show how it behaves under load. Each Candidate has its own URL, so existing evaluations can be run against it to assess output quality.

• Promoting a Candidate from the dashboard starts a production rollout at the existing Endpoint URL.
• Other Candidates remain available for independent testing.
• Endpoint Candidates use standard Modal compute pricing, including compute used for mirrored requests.

Keeping slow mirrors from blocking production

Modal says one of the hard parts of building traffic mirroring was preventing slow mirrors from holding up production. When several Candidates receive copies of a large request, they may read the data at different speeds, and a proxy can end up buffering large amounts of data while waiting for the slower ones.

To handle this, Modal extended its Pingora-based proxy to fork each request stream into a separate, bounded buffer for every Candidate receiving a copy. Each mirror reads independently. If a new chunk arrives and a buffer is full, the proxy cancels that mirrored request while production and the other mirrors continue.

A per-mirror bound alone does not cover a busy proxy that mirrors many requests at once, so Modal also caps the body data retained by mirrors within each proxy process. Mirrors share request-body chunks, so each chunk counts once against the budget. Because a network write may still hold a chunk after it leaves a buffer, the chunk keeps counting until the last mirror releases it. If adding the next chunk would exceed the budget, the proxy cancels that request's mirrors while the original request continues.

Mirroring is best-effort by design. Mirrored requests may be dropped under pressure so that production does not have to wait for a slow or failing Candidate.

What it means for users

For teams running Dedicated Endpoints on Modal, Candidates offer a way to compare a new model or checkpoint against real traffic before switching. Because mirrored requests are best-effort, some copies may be dropped under load, and mirrored compute is billed at standard rates.

Modal says it plans to add more control over compute placement and serving recipes, as well as richer metrics views for comparing deployments. It also wants to automate more experimentation. The company ties this to its earlier Auto Endpoints work on automating inference optimization, describing Candidates as a place for those optimizations to prove themselves: Modal could test promising changes against a customer's traffic and then surface measured improvements in latency, throughput or cost.