Ultralytics benchmarks TensorRT against ONNX Runtime on 19 GPUs

ONNX and TensorRT serve different roles

Ultralytics says ONNX and TensorRT are not direct rivals. ONNX, or Open Neural Network Exchange, is a model format that stores a trained model's graph and weights in a framework-neutral file. TensorRT is NVIDIA's inference engine, which typically takes an ONNX file as input and compiles it into an optimized engine.

ONNX Runtime runs ONNX models through hardware back ends called execution providers, including CPU, CUDA and TensorRT. This allows one model file to run across supported hardware. ONNX Runtime can also use TensorRT's execution provider, while CUDA can handle nodes that TensorRT does not support.

A standalone TensorRT engine, by contrast, is compiled for an NVIDIA GPU type. Ultralytics says its YOLO models can be exported to ONNX or TensorRT with a single argument.

TensorRT FP16 led the YOLO26 speed comparison

The benchmarks measured Ultralytics YOLO26 detection models on 19 NVIDIA GPUs from the Ampere, Ada and Hopper families. Testing used batch 1, 640-pixel inputs and COCO val2017, with warm-up excluded and per-run times sigma-clipped. Accuracy was measured as mAP 50-95.

TensorRT FP16 was faster than the published ONNX FP16 results on every GPU and model size, while accuracy stayed within 0.001 mAP 50-95. Median speedups ranged from 1.85x for YOLO26m to 2.55x for YOLO26n. The published accuracy values were 0.397 for both YOLO26n formats, 0.471 for YOLO26s, 0.519 for YOLO26m, 0.535 for YOLO26l with TensorRT FP16 versus 0.536 with ONNX FP16, and 0.564 for YOLO26x with TensorRT FP16 versus 0.563 with ONNX FP16.

Ultralytics cautions that the benchmark does not identify which ONNX Runtime execution provider produced the ONNX results. It therefore does not establish how ONNX Runtime compares in general, or how it compares when using its TensorRT execution provider.

INT8 can improve speed, but requires validation

The comparison also tested TensorRT INT8. The results show that lower precision is not automatically faster. On YOLO26n, INT8 was slower than TensorRT FP16 on 17 of the 19 GPUs and reduced accuracy by 0.04 mAP 50-95.

INT8 helped more often on larger models, reaching up to 1.67x the speed of TensorRT FP16 on YOLO26x on an L4. It also reduced accuracy by between 0.034 and 0.076 mAP 50-95 across the model sizes. The benchmark used NVIDIA ModelOpt with COCO128 as calibration data, while Ultralytics recommends calibrating with representative images from the deployment domain and validating accuracy on the intended data.

The deployment choice depends on the hardware

Ultralytics recommends starting with TensorRT FP16 when a YOLO26 deployment uses an NVIDIA GPU and latency matters. A TensorRT engine is tied by default to the GPU and TensorRT version used to build it, so an engine built on an RTX 4090 is not intended to be copied directly to an H100 or Jetson. The company says engines should generally be built on, or for, the target device.

ONNX Runtime is the more flexible option when one model file must run on different kinds of hardware, including CPUs, non-NVIDIA hardware or mixed fleets. The same ONNX file can run wherever ONNX Runtime and a suitable execution provider are available. Both formats support variable input sizes, although TensorRT dynamic shapes require an optimization profile defining minimum, optimal and maximum dimensions.

Ultralytics says the results apply to the tested setup, which used batch 1 and 640-pixel inputs. Users should benchmark their own model, data and deployment device before selecting a format, especially when considering ONNX Runtime with TensorRT underneath or an INT8 engine.