NVIDIA Releases Open-Source C++ Samples for Local AI With TensorRT RTX
NVIDIA has published Do Inference Now (DIN) Deploy, an open-source collection of C++ samples that pairs ONNX Runtime with the NVIDIA TensorRT RTX execution provider. According to NVIDIA, it helps developers go from a model checkpoint to a native, hardware-accelerated application on Windows and Linux.
- DIN Deploy is an open-source set of C++ samples combining ONNX Runtime with the NVIDIA TensorRT RTX execution provider.
- Samples cover Whisper, Parakeet TDT and Nemotron ASR Streaming for speech, Meta SAM 2.1 for segmentation and FLUX.2-klein-4B for image generation.
- On DGX Spark, Parakeet TDT reaches 206.41x real time on GPU versus 14.44x on CPU, and SAM 2.1 runs at 38.3 FPS on GPU versus 0.5 FPS on CPU.
- The FLUX.2 sample uses ONNX Runtime 1.25 graphics interop with Vulkan and DirectX, and shows a quantized ONNX model as a drop-in replacement.
- CMake presets cover Windows, Linux, x86-64 and Arm64.
From Hugging Face checkpoint to native C++ app
Each DIN Deploy sample begins with a Python exporter that downloads a model checkpoint from Hugging Face and converts it into an ONNX artifact. The application side is a native C++ command-line interface built on ONNX Runtime (ORT). NVIDIA says this split keeps model conversion separate from deployment logic, so developers can take an exported model into a local application without a model-specific runtime.
Most of the sample code relies on ONNX Runtime session and tensor APIs in C++. Vendor-specific code, including CUDA APIs and kernels, appears only in optional accelerated paths. Execution providers that support the required ONNX Runtime tensor APIs can run the shared code, and ORT's copy tensor API keeps data locality manageable without dedicated vendor API usage in shared code. The same ONNX Runtime API can also be accessed through WinML 2.0.
The repository provides CMake presets for Windows and Linux, including Arm64 variants. DirectX is available only on Windows.
Speech, segmentation and image generation
For automatic speech recognition, DIN Deploy supports offline and streaming pipelines. OpenAI Whisper covers offline transcription across model sizes, while NVIDIA Parakeet TDT and NVIDIA Nemotron ASR Streaming provide streaming pipelines.
Meta SAM 2.1 samples support interactive masking for images and video. They turn model outputs into segmentation masks that native applications can use for selection, tracking and other computer-vision workflows.
The FLUX.2-klein-4B sample covers prompt-driven image generation. It includes graphics-API interops with Vulkan and DirectX, which let applications integrate GPU-resident resources with a cross-vendor shader interface. For preprocessing and postprocessing around the exported model, it uses ONNX Runtime's graphics interop capability, introduced in version 1.25.
The sample also demonstrates post-training quantization (PTQ) with NVIDIA Model Optimizer, which produces a quantized ONNX model. NVIDIA notes that quantization is hardware-dependent, but because the ONNX interfaces stay unchanged, the quantized model is a drop-in replacement that requires no application-code changes.
Performance on DGX Spark
NVIDIA published GPU and CPU measurements for selected workloads on DGX Spark. Audio results are given as multiples of real time, where higher is faster.
- openai/whisper-large-v3-turbo: 58.5x on GPU, 3.8x on CPU
- nvidia/nemotron-3.5-asr-streaming-0.6b: 39.01x on GPU, 3.24x on CPU
- nvidia/parakeet-tdt-0.6b-v3: 206.41x on GPU, 14.44x on CPU
- facebook/sam2.1-hiera-base-plus: 38.3 FPS on GPU, 0.5 FPS on CPU
Getting started and what it means for developers
NVIDIA suggests starting with the repository's CMake presets for Windows, Linux, x86-64 and Arm64. After configuring and building the project, developers can export a model to ONNX and run the CLI with TensorRT RTX, or copy the code into their own application and reuse the pipeline implementations. CMake downloads ONNX Runtime and TensorRT RTX by default.
For teams building local AI features, the samples offer working reference code for several common tasks, including transcription, segmentation and image generation, in a native C++ setting. The figures come from NVIDIA's own tests on DGX Spark, so results on other hardware may differ.
NVIDIA Developer: Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples
