NVIDIA TensorRT Edge-LLM Runs Nemotron-3.5 Lightning on DGX Spark in a Few Commands

One runtime from development to deployment

NVIDIA says AI is moving beyond the data center as developers build agents for the edge. The examples it names include monitoring driver attention and cabin conditions, helping robots understand their surroundings and plan tasks, and coordinating fleets of machines on production lines. Such applications must work within tight memory, power and latency budgets, and NVIDIA notes that getting a model ready to run can involve substantial setup.

NVIDIA DGX Spark, now available in a 64GB configuration, is positioned as a platform where embedded developers can prototype, fine-tune and test models and agents. TensorRT Edge-LLM is a lightweight C++ inference runtime designed for embedded workloads. According to NVIDIA, it brings the same inference framework to development on Spark and to deployment on platforms such as Jetson Thor, DRIVE AGX Thor and IGX Thor.

From install to chat and API server

The tutorial centers on the trtedge command-line interface, which downloads a chosen model and builds TensorRT engines optimized for inference on Spark. The example model is NVIDIA Nemotron-3.5 Lightning, a text-only model.

Requirements are a DGX Spark with a supported NVIDIA software stack, NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10 to 3.12. The examples were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11. Users create a virtual environment, install the package, verify the TensorRT version, and then start a chat with the trtedge chat command.

The first launch requires a model download of approximately 21GB and an engine build. Files are stored in ~/.cache/tensorrt-edgellm by default, so users need space for both the model and the engines. Later launches reuse matching cached engines.

The same model can also be served through a local OpenAI-compatible API on Spark with trtedge serve, intended for application development and testing. A health endpoint confirms readiness, and the chat completions endpoint streams responses as JSON events.

DSpark speculative decoding and benchmark results

As an optional step, developers can enable DSpark speculative decoding. A draft model proposes several tokens, and the main model verifies them in a single pass. The tutorial uses three draft tokens and notes that users can experiment with the number of speculative tokens and other settings to tune speed for their workload.

NVIDIA reports the following measurements on DGX Spark 64GB, with median time to first token (TTFT) and mean generation speed:

• SpeedBench 1K: 201.0 ms and 73.3 tokens/s with DSpark off; 203.0 ms and 108.5 tokens/s with DSpark on.
• SpeedBench 2K: 329.5 ms and 72.8 tokens/s with DSpark off; 331.6 ms and 105.3 tokens/s with DSpark on.
• HumanEval prompts: 98.1 ms and 73.4 tokens/s with DSpark off; 98.7 ms and 126.9 tokens/s with DSpark on.

Peak total system memory use during inference with DSpark was 27.9GB, including the OS. NVIDIA states the tests ran on DGX OS 7.6.0 (Ubuntu 24.04.4 LTS) using NVFP4 with thinking enabled, batch size one, one concurrent streaming request and a 128-token output limit. Average input lengths were 1,153 tokens for SpeedBench 1K, 2,247 for SpeedBench 2K and 139 for HumanEval.

What it means for edge developers

The tutorial lowers the effort needed to try a model on Spark: a few commands give a terminal chat and a local API that mimics the OpenAI interface, which lets teams prototype applications before moving to target hardware. In NVIDIA's measurements, DSpark left time to first token almost unchanged while raising generation speed across all three workloads.

NVIDIA suggests trying other models from the supported models list, including vision-language models that interpret images and video, as well as speech recognition and text-to-speech models. Developers should check the TensorRT Edge-LLM documentation for model-specific workflows and requirements on their target embedded platform.