vLLM Adds Day-One Support for Thinking Machines’ Inkling 2 Small

vLLM has added day-zero support for Thinking Machines’ Inkling 2 Small, a multimodal mixture-of-experts model that accepts text and images and produces text. The 48-layer model has 594B total parameters, 17B active parameters and a context window of up to 1M tokens.
- vLLM supports Thinking Machines’ Inkling 2 Small on day zero.
- The multimodal MoE model has 594B total and 17B active parameters.
- Its context window reaches up to 1M tokens.
- Native DSpark drafts seven tokens per step.
- vLLM reports up to 630 tokens per second for a single user at TP4.
A larger multimodal model with a revised architecture
Inkling 2 Small uses 384 routed experts with top-8 routing across all layers. Its 48 layers include 40 sliding-window attention layers and eight full-attention layers, while the model uses a 1,024-token sliding window.
The model retains parts of the original Inkling backbone, including grouped-query attention with learned relative bias, four short convolutions per layer and a sigmoid-routed MoE design. It also introduces block attention residuals, gated attention with a per-token attention sink, and a 23-layer ViT image encoder operating on 20×20 patches.
Compared with Inkling, the new model accepts text and images rather than text, images and audio. vLLM says the model can be served on a single node because its mixed-precision MoE uses NVFP4 in layers 0, 39 and MXFP8 in layers 40, 47.
Native DSpark support and vLLM optimizations
Inkling 2 Small ships with a native DSpark speculator. Its five-layer RNN head drafts seven tokens per step, replacing Inkling’s eight chained MTP heads. vLLM supports DSpark speculative decoding and also provides reinforcement learning support through SkyRL, along with LoRA support.
The integration combines kernel fusion, programmatic dependent launch and sequence parallelism. vLLM says its fused kernels combine communication and memory-bound operations around the model’s attention residuals, while sequence parallelism removes repeated work during prefilling. On an 8K-token prefill, this cuts the step by 22% at TP4 and 26% at TP8, according to the company, and frees 1.3, 1.9 GiB per GPU for KV cache.
vLLM also developed a warp-specialized Gluon kernel for decode. In its test, a batch-1 decode step at 100K context fell from 15.1 ms to 4.8 ms. The company attributes the change to packing six attention heads into one tile and reading the key and value data once.
Reported throughput and accuracy checks
In an agentic workload using TP8, DSpark with probabilistic drafting and prefix caching, vLLM measured output throughput per GPU at a concurrency where each user received 100 tokens per second. At 32K context, the result was about 1,070 output tokens per second per GPU across about 84 concurrent sessions, with a 91% prefix-cache hit rate and 3.3 acceptance length.
At 64K, 128K and 256K context, the reported throughput was about 915, 700 and 450 output tokens per second per GPU, respectively. The corresponding concurrent session counts were about 72, 55 and 37. For a single user at TP4, vLLM reports about 630 tokens per second with an acceptance length of 3.5.
vLLM says its implementation preserves the reference numerics, including FP32 attention output and router calculations and the same NVFP4 and MXFP8 activation quantization. The team verified accuracy with DSpark enabled on NIAH up to 1M tokens, MMMU-Pro and HLE, and says the scores matched runs without speculative decoding.
vLLM: Inkling 2 Small on vLLM: Day-0 Support


