NVIDIA Cosmos Uses Two Prompts to Build Physical AI Datasets

NVIDIA has outlined a two-prompt workflow in NVIDIA Cosmos for building training-ready datasets for physical AI systems. The workflow uses coding agents to configure cloud-native blueprints, generate or augment data, and coordinate labeling and evaluation.
- NVIDIA Cosmos Data Factory includes five blueprints and six composable skills.
- A first prompt configures the environment, while a second runs a training-data workflow.
- Use cases include autonomous driving, video augmentation, rare events and defect images.
- NVIDIA reports up to 60x faster dataset assembly and 45x cheaper AV scenario generation.
Two prompts connect setup and data generation
The workflow is built around NVIDIA Cosmos Data Factory, which NVIDIA describes as a collection of five cloud-native blueprints and six composable skills for data curation, processing and synthetic data generation. The platform is part of NVIDIA Cosmos, an open platform of frontier models, datasets, frameworks and skills for world model development.
The first prompt prepares the environment: “Clone the CDF Orchestration repository, then run orchestration preflight and setup.” According to NVIDIA, preflight checks the environment without making changes, while setup prepares it to run the blueprints. The CDF Orchestration core skill coordinates workflow steps, deploys required services and manages execution on a cluster.
The second prompt runs a blueprint for a selected training-data task. NVIDIA says frontier AI agents can orchestrate the blueprints, enabling prompt-to-data workflows without human steering.
Five use cases target physical AI training gaps
NVIDIA presents five reference use cases. They cover generating driving scenarios for autonomous vehicle training, augmenting image attributes for search and re-identification, augmenting videos across weather, lighting and environments, generating rare-event videos, and generating defect images for visual inspection.
For autonomous vehicle data, the relevant blueprint searches databases of driving scenarios, retrieves related videos and transforms them into conditions such as rain, snow and fog. NVIDIA says Cosmos Evaluator checks the generated videos for physical plausibility. An example prompt asks the system to search for similar driving scenes, apply weather changes while preserving layout and motion, and check the result.
Other workflows can create controlled clothing variations in person images, transform existing footage, or generate events such as fires, falls and shoplifting from a source image. The generated results can be checked for requested conditions, motion artifacts and physical plausibility before being auto-labeled with detections, tracks, captions or semantic annotations.
These capabilities address cases where data is difficult, expensive or dangerous to capture. NVIDIA specifically cites rare defects, nighttime incidents and other challenging scenarios as examples of gaps in physical AI datasets.
NVIDIA reports faster dataset assembly
NVIDIA says its measurements on a cloud server with eight NVIDIA RTX PRO 6000 Blackwell Series GPUs showed up to 60x faster dataset assembly and 45x cheaper autonomous vehicle scenario generation compared with traditional methods. These are NVIDIA-reported results, rather than independent benchmarks described in the post.
The company is positioning Cosmos Data Factory for developers working on autonomous vehicles, robotics and vision AI agents. Its composable skills, including Auto Labeling, AnomalyGen, Augmentation, Curation and Retrieval, Simulation and Orchestration, can be used separately or combined into custom workflows.
For users, the announcement points to a way to move from existing images or videos to larger, more varied and labeled training datasets through agent-directed workflows. NVIDIA directs developers to its Cosmos Data Factory blueprints and composable skills, and to a livestream about building training-ready datasets in two prompts.
NVIDIA Developer: Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos


