Artificial intelligence news.
Straight from the original announcements.

AWS shows how to route coding agent requests with NeMo Switchyard

November 13, 2026, 23:30By the AI News Bites Editorial TeamSource: AWS
AWS shows how to route coding agent requests with NeMo Switchyard
Image: AWS

AWS has published a sample deployment that routes coding agent requests between models on Amazon Bedrock using NVIDIA NeMo Switchyard and LiteLLM. The approach measures performance by cost per solved task, rather than by token price alone.

Key points
  • AWS uses cost per solved task to evaluate coding agent routing.
  • NVIDIA NeMo Switchyard supports Stage and Random routing in the LiteLLM integration.
  • The sample deploys LiteLLM and Switchyard on Amazon ECS with AWS Fargate.
  • LiteLLM routes requests to models through Amazon Bedrock.
  • AWS labels the integration experimental and intended for non-production use.

Why task-level costs matter

Coding agents such as Claude Code and OpenAI Codex CLI may make a few model calls or dozens while completing one developer request. They can read files, edit code, run tests and retry work, meaning the total cost depends on the number of turns, the history replayed in each turn and whether the task finishes successfully.

AWS says developers pay for every attempt, including work on tasks that never finish. It therefore recommends evaluating routing by cost per solved task: the total cost of runs divided by the number of runs whose tests pass. This gives teams a way to compare a routed model group with individual model baselines on their own workloads.

How Switchyard selects a model

NVIDIA NeMo Switchyard is an open source library that lets an AI agent choose which model handles each request. In the sample's LiteLLM integration, Stage routing examines recent tool activity and results in the request history before choosing between two candidate models. The agent must send the full conversation history for this strategy to work.

Random routing instead selects from configured models, with optional weights. AWS describes it as useful for traffic splits and A/B comparisons. Stage requires exactly two candidates, while Random requires at least two. Neither strategy uses a judge model. Classifier-backed strategies such as Task and Composite routing require intermediate model calls that this integration does not support.

The AWS deployment architecture

The walkthrough uses the AWS Cloud Development Kit to deploy LiteLLM, an open source gateway for large language models, with the Switchyard routing plugin in an AWS account. LiteLLM calls models through Amazon Bedrock. LiteLLM and the plugin run together in one container on Amazon Elastic Container Service with AWS Fargate, using private subnets.

Requests use an OpenAI-compatible interface with the model field set to switchyard. AWS Certificate Manager, Amazon Route 53, AWS WAF and an Application Load Balancer provide the endpoint and network controls. The load balancer forwards only /v1/chat/completions, /v1/models and /health/liveliness to the ECS task. LiteLLM then calls the selected model through the Amazon Bedrock Converse or ConverseStream API.

The response includes an x-litellm-model-name header identifying the model that answered, while the response body's model field remains switchyard. Inference traffic uses an Amazon Bedrock Runtime interface endpoint in the Amazon VPC.

A sample for evaluation, not production

The sample requires access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B in Amazon Bedrock. AWS says the pair has been tested in US East (N. Virginia), and asks users to check regional availability and tool-calling support elsewhere.

AWS also cautions that the sample installs switchyard-server 0.3.0 and uses the published LiteLLM v1.102.0 image. It describes switchyard-server 0.3.0 as a demo and evaluation component, and says the experimental integration's configuration and APIs may change. The company recommends deploying in a non-production account, testing before reliance and working with security and legal teams on organizational requirements.

For users building coding agents, the example provides a concrete way to test whether routing reduces the cost of completed work. Its central measure, however, is tied to each team's workloads, model configuration and test outcomes, rather than to a token price considered in isolation.

Related news

AWS Says Agents Wrote Its 200+ Service Emulator Without Human Code Review
For developersNovember 12, 2026, 22:52

AWS Says Agents Wrote Its 200+ Service Emulator Without Human Code Review

AWS says autonomous agents wrote and tested every line of AWS Emulator, a free local copy of more than 200 AWS services, while a merge gate controlled which changes…

AWS adds models, agent safeguards and cost controls in October
Products and toolsNovember 11, 2026, 23:26

AWS adds models, agent safeguards and cost controls in October

AWS says its October 2026 releases give AI builders more model choices, stronger controls for production agents and clearer ways to manage costs. The updates span Amazon…

Moonshot AI’s Kimi K3.1 Pro Highspeed arrives on Amazon Bedrock
ModelsNovember 11, 2026, 02:07

Moonshot AI’s Kimi K3.1 Pro Highspeed arrives on Amazon Bedrock

Moonshot AI’s Kimi K3.1 Pro Highspeed is now available on Amazon Bedrock, giving developers access to a post-trained Kimi K3 model with a 1-million-token context window…