NVIDIA Cosmos Framework adds agent skills and simulation recipes for physical AI

Two problems, two sets of tools

According to NVIDIA, physical AI developers working with world models face two challenges: adapting models to their sensors, tasks and data, and getting those models into applications reliably. Cosmos is meant to address both.

Cosmos Framework provides agent skills and recipes for world model training, multimodal data processing, optimization and inference. Cosmos NIM microservices handle GPU-accelerated serving through application APIs. The post walks through a simulation workflow and a deployment workflow, both starting from released checkpoints.

NVIDIA describes the framework as open and fully modular. It is expanding beyond Cosmos 3 to support NVIDIA Alpamayo, GR00T, Qwen, pi0.5 and other compatible Hugging Face models.

Training, reinforcement learning and optimization

For training, the framework supports text, images, video, audio and actions. Developers can use LoRA for parameter-efficient adaptation or full supervised fine-tuning (SFT) to update all model parameters. Data can be loaded through JSONL, WebDataset, LeRobot or custom adapters.

The platform now covers online and offline reinforcement learning. Newly supported pipelines include:

• DPO: offline preference optimization.
• GRPO: online RL with a rule or model reward and vLLM rollouts.
• PPO: actor-critic RL with frozen reference and critic, including RL for embodied policies in simulation.
• OPD: on-policy distillation from a frozen teacher.
• DiffusionNFT: online RL for diffusion world models.

Rollout, reward and training roles can run on shared GPUs or separate pools, and supported simulators can be connected to collect experience. For optimization, models can be distilled to reduce size or generation steps and quantized to reduce memory use. Compatible checkpoints run with Hugging Face Diffusers, PyTorch, SGLang, vLLM-Omni, vLLM, TRT-LLM, TRT-Edge-LLM or NVIDIA NIM.

The framework also ships setup, inference, post-training and troubleshooting skills under .agents/skills, so a coding agent can be given a bounded task such as post-training a checkpoint on a dataset with a set GPU-hour limit.

Cosmos 3 Nano simulation recipes for vehicles, robots and surgery

Cosmos Framework includes post-training and inference recipes for robot simulation, realistic scene generation, multi-camera driving simulation and surgical policy evaluation. The new Cosmos 3 Nano recipes and example checkpoints let developers adapt the model to custom sensors, action spaces and data domains. The checkpoints support PyTorch and vLLM-Omni and were tested on NVIDIA Hopper and Blackwell GPUs, according to NVIDIA.

Both autonomous vehicle checkpoints take a prompt and World Scenario Map (WSM) controls representing lanes, objects, road markings, traffic signs and road boundaries. Prompts can describe weather, lighting, time of day and road conditions.

Cosmos3-Nano-Transfer-Auto uses bidirectional generation to denoise an entire multi-view clip at once, aimed at offline synthetic data generation. It supports 30 fps input and output video, with up to 400 frames at 480p or 720p. Longer sequences can be generated through autoregressive chunking. Changing the prompt can turn a daytime drive into a rainy night while keeping the same road layout and vehicle trajectories, which NVIDIA suggests using to train and evaluate perception and driving models across conditions.

The reference run on four GPUs produces seven 720p videos (297 frames at 30 FPS) and 99 LiDAR sweeps. NVIDIA advises checking the output files and verifying WSM alignment and consistency across camera views and LiDAR.

Cosmos3-Nano-Sim-Auto is the causal, few-step-distilled variant, enabling autoregressive streaming generation. It produces observations incrementally as an AV policy or human driving input updates the scenario through a WSM renderer. The generated camera and LiDAR observations return to the policy, which enables closed-loop evaluation.

Deployment and what it means for developers

On the serving side, Cosmos NIM microservices provide accelerated inference with quantization, speculative decoding and KV-cache-aware routing. NVIDIA says this achieves roughly 2, 3x speedups on H200 GPUs. GPU memory needs vary by model and task: Cosmos3-Nano generation needs 58 GiB in BF16 and 44 GiB in FP8 with offloading.

For teams building autonomous vehicles, robots or surgical systems, the release offers released checkpoints and runnable recipes for generating synchronized camera and LiDAR data, plus a path to closed-loop testing. NVIDIA also points developers to a full technical report and a livestream on Thursday, Nov. 5 on using Cosmos Framework.