Artificial intelligence news.
Straight from the original announcements.

AWS Makes SGLang-Neuron Publicly Available for Trainium

November 12, 2026, 23:03By the AI News Bites Editorial TeamSource: AWS
AWS Makes SGLang-Neuron Publicly Available for Trainium
Image: AWS

AWS says SGLang-Neuron is now publicly available through an AWS Deep Learning Container released by RadixArk in collaboration with Annapurna Labs. The plugin brings SGLang model serving and reinforcement learning rollouts to AWS Trainium while preserving SGLang’s existing interface.

Key points
  • SGLang-Neuron is publicly available through an AWS Deep Learning Container.
  • The plugin brings SGLang serving and RL rollouts to AWS Trainium.
  • Native PyTorch operations and Neuron Kernel Interface kernels run the execution path.
  • PD disaggregation transfers the prompt KV cache between servers over NIXL and EFA.
  • AWS’s walkthrough uses a Trainium3 EC2 instance and the openai/gpt-oss-120b model.

SGLang workflow reaches AWS Trainium

SGLang-Neuron lets users serve models and generate reinforcement learning, or RL, rollouts with SGLang on AWS Trainium. According to AWS, users can launch models through SGLang’s server interface and invoke the endpoint with the same generation APIs used in existing SGLang workflows.

Trainium is AWS’s portfolio of AI chips for training and inference at scale. AWS says the SGLang integration runs on native PyTorch beneath the familiar serving interface. Its out-of-tree plugin design allows Trainium support and the SGLang core to develop independently.

How the Trainium integration works

SGLang-Neuron connects SGLang with Trainium through a plugin. Installing it registers the Neuron platform through SGLang’s plugin entry points, while setting SGLANG_PLATFORM=neuron selects the platform when the server starts.

SGLang core continues to provide the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin handles Trainium-specific models, attention, mixture of experts, or MoE, and the graph runner.

TorchNeuron registers Trainium as a native PyTorch device. SGLang-Neuron uses standard PyTorch operations for the model execution path, alongside kernels written with the Neuron Kernel Interface. It compiles those operations and kernels with torch.compile, then prepares and executes them through the graph runner during serving.

Prefill and decode can be separated

SGLang-Neuron also supports prefill and decode disaggregation, known as PD disaggregation. In this setup, a prefill server computes the prompt’s key-value cache and transfers it to a decode server using NIXL over Elastic Fabric Adapter, or EFA.

The decode server generates the remaining tokens. AWS says both servers continue to use SGLang’s scheduler and request lifecycle, while the plugin manages Trainium execution and the key-value cache transfer path.

Getting started with the public container

AWS’s walkthrough uses a Trainium3 Amazon Elastic Compute Cloud, or Amazon EC2, instance with a pytorch-inference-sglang-neuronx container. Users start the container, launch the SGLang model server, and send a request to its chat completions endpoint.

The example serves openai/gpt-oss-120b with tensor and expert parallelism settings, an FP8 key-value cache, a context length of 10240, and a maximum of 64 running requests. AWS directs users to the SGLang-Neuron getting started guide and repository documentation for setup instructions, supported models, features and validated configurations.

For users already working with SGLang, the announcement means they can apply the same server and generation API workflow on Trainium. AWS is also inviting users to try the project with their workloads, contribute through its repository and share feedback on future model and feature support.

Related news

AWS shows how to route coding agent requests with NeMo Switchyard
For developersNovember 13, 2026, 23:30

AWS shows how to route coding agent requests with NeMo Switchyard

AWS has published a sample deployment that routes coding agent requests between models on Amazon Bedrock using NVIDIA NeMo Switchyard and LiteLLM. The approach measures…

AWS adds three-tier Data Capture to SageMaker HyperPod
Products and toolsNovember 13, 2026, 19:01

AWS adds three-tier Data Capture to SageMaker HyperPod

Amazon SageMaker HyperPod now includes Data Capture, a built-in capability that records live inference inputs and outputs to Amazon S3. AWS says the feature is designed…

Moonshot AI’s Kimi K3.1 Pro Highspeed arrives on Amazon Bedrock
ModelsNovember 11, 2026, 02:07

Moonshot AI’s Kimi K3.1 Pro Highspeed arrives on Amazon Bedrock

Moonshot AI’s Kimi K3.1 Pro Highspeed is now available on Amazon Bedrock, giving developers access to a post-trained Kimi K3 model with a 1-million-token context window…