Rafay adds budget-based routing for frontier and open-source AI models

One endpoint for multiple model providers

Token Router sits between applications and the locations where inference runs. It can send requests to in-cluster deployments or external providers such as OpenAI, Anthropic and any OpenAI-compatible endpoint, using priority, provider health and cost as routing factors.

Applications use one API and one set of credentials, while operators can change the model source without requiring code changes. Requests can be sent to a highest-priority provider group first and then moved to another group if the primary is unavailable. Rafay cites configurations such as self-hosted models with cloud bursting, frontier models with in-cluster fallback, and multi-vendor routing.

Traffic is balanced across healthy providers in a group. Providers that fail health checks are removed from rotation automatically and return when they recover. Rafay also says operators can set separate rates for input, output, cached input and cache write tokens, with each request metered according to the group that served it.

Budgets can shift traffic instead of simply blocking it

Token Budgets let administrators set daily or monthly spending limits for an organization, project or individual user, using the currency supported by the service provider. A user's budget follows them across projects, and every request is checked against applicable budgets before admission.

When a limit is reached, an operator's policy can deny requests, continue serving them or fall back to a lower-cost model. Token Router can also move traffic to a lower-cost or self-hosted model when a tenant, team or API key approaches or reaches its budget. This lets users continue receiving answers while spending remains within the administrator's limit.

New projects and users inherit default budgets, while administrators can apply overrides for heavier workloads. Threshold alerts notify administrators before limits are reached. Developers can view their spending and remaining budget in Rafay Developer Hub, and budgets reset automatically each day or month.

Targeting neoclouds and enterprise AI teams

Rafay says the capabilities are designed for neoclouds offering open-source models as a service and enterprises operating internal token factories. A neocloud can add frontier models to its catalog through the same API and credentials, offer a daily allowance for frontier usage, then route additional traffic to open-source models running on its own GPUs.

For enterprises, a platform team can set a daily frontier quota for a team or application. Once that quota is used, requests can move to an open-source model on company GPUs, while applications continue calling the same endpoint. Rafay says a token usage dashboard with per-user filtering and CSV export supports internal showback and chargeback.

“Token Router and Token Budgets turn that tradeoff into policy,” Mohan Atreya, chief product officer at Rafay Systems, said in the announcement. Rafay also added support for LiteLLM Generic Guardrail API providers, separate cached input token pricing, and API key rotation policies with expiration, overlap windows and alerts.