NVIDIA Nemotron 3.5 Super VL arrives on Amazon EKS and SageMaker HyperPod EKS
AWS has published a guide to deploying NVIDIA Nemotron 3.5 Super VL, a 120B-parameter open multimodal model, on Amazon EKS, EKS Auto Mode and SageMaker HyperPod EKS. The model activates 12B parameters per token and handles text, image, document and video input.
- Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters per token and up to 1M tokens of context.
- It accepts text, image, document and video input and outputs text; NVIDIA publishes weights, datasets and post-training recipes.
- AWS's reference implementation runs on Amazon EKS, EKS Auto Mode and SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM.
- The architecture was validated on p6-b200.48xlarge and p5en.48xlarge instances.
- In AWS's example, one replica serves about 320 tokens per second within a TTFT under 600 ms and ITL under 100 ms.
What Nemotron 3.5 Super VL offers
According to AWS, Nemotron 3.5 Super VL joins the Nemotron 3 Nano, Nano Omni, Super, Ultra and Nemotron 3.5 Lightning models already available on its platform. It is a mixture-of-experts (MoE) model with 120B total parameters, of which 12B are active per token. It combines reasoning and tool use with text, image, document and video understanding, so a single model can serve both text and visual tasks.
The model is distilled from Nemotron 3 Ultra. NVIDIA publishes the weights, training datasets and post-training recipes, which lets teams customize it for domain-specific workloads and run it on infrastructure they control. Input can be text, image, document or video, and output is text. The context length reaches up to 1M tokens, and BF16 and NVFP4 checkpoints are available.
NVIDIA reports 63.44 percent on Hermes Bench, 87.05 percent on LangChain agent evaluations and 80.8 percent on the CharXiv chart-reasoning benchmark.
Architecture and use cases
The model uses a hybrid Mamba-Transformer MoE design that interleaves Mamba-2, attention and expert layers, with a RADIO vision encoder. Because sparse routing activates only 12B of the 120B parameters per token, compute per token matches a much smaller model, while GPU memory still has to hold the full weights. Mamba-2 layers carry a constant-size state during generation, so their memory does not grow with sequence length the way KV cache does. AWS says this keeps long contexts affordable. A recursively shared Multi-Token Prediction (MTP) head supports native speculative decoding.
The listed use cases are agent workflows that mix reasoning with visual understanding:
- Coding: test and validate code, analyze errors, recommend fixes.
- Web research: cross-reference and verify sources returned by search tools.
- Document intelligence: read tables, charts and page layouts to produce summaries.
- Cyber security: identify vulnerabilities and triage threats.
- Video understanding: search, content, review and incident analysis.
The model integrates with major open agent frameworks through NVIDIA reference implementations, cookbooks and the NVIDIA Agent Toolkit.
For vLLM, the minimum GPU counts differ by precision. With BF16, B200/GB200 needs 8 GPUs, B300/GB300 needs 4, H200 needs 8 and H100 needs 16. With NVFP4, the counts are 2, 1, 4 and 8 respectively.
Deployment on AWS
The reference implementation, part of the aws-do-eks project, runs on standard Amazon EKS, EKS Auto Mode and SageMaker HyperPod EKS. The Kubernetes manifests are the same for each orchestrator option. With EKS Auto Mode, AWS manages scaling, GPU and EFA support in the cluster. Each cluster has a control plane and a data plane of CPU and GPU nodes: CPU nodes run system and test pods, while GPU nodes run the model. Amazon ECR stores container images, a shared Amazon FSx for Lustre volume holds the weights, and Elastic Fabric Adapter (EFA) connects the GPU nodes.
NVIDIA Dynamo serves the model either aggregated on a single node or disaggregated, with prefill and decode on different nodes. For vision-language models, the image and video encoding stage can also run on its own node.
AWS says the architecture has been validated on p6-b200.48xlarge instances (8 x NVIDIA B200 GPUs) and p5en.48xlarge instances (8 x NVIDIA H200 GPUs). Users can check the cluster and network with an NCCL all-reduce micro-benchmark, and measure the deployed model with NVIDIA's aiperf tool, which reports time to first token (TTFT), inter-token latency (ITL) and throughput in tokens per second.
Planning capacity from concurrency sweeps
AWS shows a concurrency sweep comparing aggregated and disaggregated deployments on two nodes. It found lower TTFT and about the same ITL for the disaggregated setup. The post suggests drawing latency budgets as horizontal lines, for example TTFT under 600 ms and ITL under 100 ms per token. The rightmost point under both lines gives the throughput one replica can serve within the service level objective. In the example that is c32, or about 320 tokens per second.
To size a deployment, divide the target tokens per second by the per-replica throughput to find the number of prefill-decode pairs needed. Rising ITL points to adding decode capacity, while rising TTFT points to adding prefill capacity. Re-running the sweep after changes to precision, parallelism shape or a new Dynamo or vLLM release shows the trade-off right away.
What it means for users
For teams that want an open multimodal model under their own control, the post offers a tested path to run Nemotron 3.5 Super VL on managed Kubernetes services in their own AWS account. The same pattern can be applied to other models in the NVIDIA Nemotron family, and more inference blueprints are available in the AI on EKS project.
