Cline hits 11,840 tokens per second on one GPU with Nemotron 3.5 Super VL

From 6,058 to 11,840 tokens per second

NVIDIA released Nemotron 3.5 Super VL on October 15. According to Cline, the model is designed to be a fast, locally hosted option that performs reasonably well across areas such as document intelligence, coding and financial accounting. Cline says it is much faster than other models in the same category, and that its small size makes it practical to post train for special use cases.

Through a collaboration with NVIDIA, Cline got access to an NVIDIA DGX Station, which has a 252 GB GPU memory budget. The starting point was the base vLLM configuration (version 0.32.0rc1.dev190+g3f1bcfd82), which used 67 GB of memory. Even there, Cline measured 6,058 aggregate tokens per second on a test with 512 input/output tokens and 256 concurrent streams. The company notes that the model's base configuration is already tuned for high throughput inference.

The five tuning steps

Cline lists five changes it applied to raise throughput:

• MTP decoding: The team used the MTP decoding head that ships with the model. Using MTP3 brought roughly 7,000 tokens per second. It used the OMP1 config, since multiple threads add coordination overhead.
• Async scheduling: With many concurrent outputs, the GPU does heavy kernel work while the CPU does heavy scheduling work. Asynchronous scheduling keeps the pipeline from being blocked while the GPU processes.
• Mamba kernel back to FlashInfer: The model uses KV cache for attention layers and recurrent state plus short convolution history for Mamba layers. Switching the Mamba kernel to FlashInfer lifted throughput to 7,800 tokens per second. Cline says it is not clear why this worked and that it may not work on every GPU configuration.
• Lookahead from MTP 3 to MTP 1: Predicting more draft tokens does not necessarily help, because they must be verified and failed verification wastes model time. With 256 concurrent threads the GPU already has plenty of useful work, so the extra overhead was not worth it. This raised throughput to 9.5K.
• Mamba state from FP32 to FP16: Cline switched the Mamba state to FP16 with stochastic rounding. This lowers precision and could reduce eval scores, but the team judged it worthwhile given its goal of maximizing throughput.

The final result was 11,840 aggregate output tokens per second.

Caveats and Cline's predictions

Cline stresses that inference engineering is highly contextual and that scaling depends on the traffic shape. Choices such as MTP 1 over MTP 3 or the FlashInfer kernel worked for this load test but, the company says, could just as easily hurt throughput under other loads. The FP16 change could also lower eval scores. Its advice is to identify the best traffic shape and tune inference for that shape alone.

The company also made three longer term predictions. First, people will be able to self host and run dozens of parallel local agentic threads. Cline points to a video of 32 parallel Clines running Nemotron 3.5 Super VL on a single DGX Station, which it describes as real footage with no speedup.

Second, running self hosted models will be cheaper than paying for inference. Cline's napkin math comparing API usage with a fully occupied GPU came to a 66% discount on inference, provided the GPU is fully utilized.

Third, autoscaling will become agentic. Cline imagines an agent with access to Grafana metrics that runs load tests on a held out GPU, finds the best configuration for the traffic hitting an endpoint, and then safely migrates traffic to it.

What it means for self-hosting

The results are Cline's own measurements on one specific setup, a single GB300 with 256 concurrent requests, and the company itself warns they may not carry over to other workloads. Still, the walkthrough shows how much throughput can be gained from configuration choices alone on an open weight model. For teams considering self-hosting, Cline's message is that savings depend on keeping the GPU fully occupied and on matching the configuration to real traffic.