Anyscale: Self-Hosted Coding Agents on Ray and vLLM Cost a Fraction of API Pricing

What Anyscale is proposing

Anyscale published a step-by-step guide for serving open-weight models to coding agents such as Claude Code, Codex and Cursor. The argument is twofold. Coding agents repeatedly send repository context, instructions, tool results and conversation history to a model, so source code and internal documents leave a company's perimeter with every request. And paying closed-source providers by the token means a bill that grows linearly per developer and per request.

According to Anyscale, owning the runtime keeps code, traces and usage data in-house, and that data could later be used to post-train models for a specific team. The company also cites BMW, which it says found the cost-benefit of moving from API to in-house inference with as few as 8 developers.

The cost model and the performance figures

Anyscale's cost model assumes roughly $4 per GPU-node hour for an RTX PRO 6000 and 50 registered developers sharing one GPU. On those assumptions it reports:

• Always-on serving: about $2,920 per month, or about $58 per registered developer.
• Work-hours-only serving: about $840 per month, or about $17 per registered developer.
• API baseline: an active developer routinely runs up about $800 per month in equivalent Claude Code API costs.

The company stresses these are planning projections rather than a total cost of ownership analysis.

Anyscale also lists results from its own evaluations:

• 8.0x faster compilation with compile-cache restoration, from 48.5 to 6.0 seconds in a no-MTP NVFP4 setup.
• 3.79x higher request throughput with CPU KV-cache tiering. In a single-replica GLM-5.2 evaluation, a 512 GiB CPU tier raised the cached share of prompt tokens from 6.1% to 74.9% across 58 common prompts and cut median time-to-first-token from 48.4 to 9.2 seconds.
• 1.86x higher output throughput with speculative decoding, as multi-token prediction raised output from 65 to 121 tokens/s.
• 2.87x faster output generation with CUDA graphs, with client-observed throughput rising from 15.9 to 45.6 tokens/s compared with eager mode.

Why coding agents need different serving

Anyscale describes coding-agent traffic as prefill-heavy. Each turn re-sends system instructions, repository context, tool definitions, diagnostics and history, often to produce only a short tool call or edit. Because most of that context repeats between turns, automatic prefix KV-cache reuse offers large gains. Correlated bursts across a team add pressure, so the company says a platform must balance low time to first token, smooth token streaming, high cache reuse and elastic capacity.

The stack has three layers. vLLM is the inference engine that loads the model on GPUs and generates tokens. Ray Serve LLM scales, routes and load-balances requests across vLLM replicas behind one OpenAI- and Anthropic-compatible endpoint. The Anyscale Platform provisions GPUs, autoscales nodes including scale-to-zero, and adds zero-downtime rollouts, fault tolerance, logs, tracing and alerting.

Each client uses a different API path, which the serving layer must expose: Cursor uses /v1/chat/completions, Claude Code uses /v1/messages and Codex uses /v1/responses.

Deployment path and what it means for teams

Anyscale released a code repository, LLM serving for coding agents, split into five parts: deploy a baseline service, connect Claude Code, Codex and Cursor, optimize GPU usage, caching, decoding and autoscaling, route between the open model and Claude with LiteLLM, and configure zero-touch deployment for a team.

The recommended deployment has four steps: select a model, select hardware and a parallelism strategy, configure the vLLM engine and Ray Serve deployment, then deploy and validate each client. On model choice, Anyscale names the Qwen, DeepSeek, Kimi and GLM families as increasingly aimed at coding and tool use. It cautions that public rankings are only a starting point, and that benchmark scores often reflect the full agent system rather than the model alone, so results should be compared only when versions and setups match.

For engineering teams weighing API spend against data control, the company's pitch is that shared GPUs can lower marginal token cost when utilization is high enough to offset fixed infrastructure expense. The figures come from Anyscale's own model and depend on its stated assumptions.