Furiosa SDK 2026.4 boosts RNGD tokens per watt, adds tiered KV caching

Efficiency gains concentrated on MoE models

FuriosaAI says SDK 2026.4 is its fourth major software release this year for RNGD and is available through its Developer Portal. The company reports more tokens per watt across 18 models relative to SDK 2026.3.

The biggest improvements, according to FuriosaAI, are on MoE models:

• GPT-OSS-120B: +138%
• GPT-OSS-20B: +57%
• Qwen3-30B-A3B: +43%

FuriosaAI also states that RNGD is now in mass production with TSMC. It argues that lower cost per token and drop-in software execution make the platform a strong option for enterprises that want to run advanced AI workloads in existing air-cooled data centers.

Two further changes target decoding speed. Profile-guided decode batching, which uses measured NPU execution and I/O costs to optimize decode batches and bucket selection, is now enabled by default for generative models. In GPT-OSS-120B benchmarks (TP=32, DP=1), FuriosaAI reports that it raised ShareGPT output-token throughput by a median 22.7% and cut mean time per output token by a median 16.0% across 20 matched configurations. Results vary by workload, and the first model load may involve 1 to 2 minutes of profiling. The SDK also introduces improvements to token sampling, which sits on the critical path for time per output token.

Experimental hierarchical KV cache

Prefix caching avoids repeated prefill for long shared contexts, but a conventional cache is limited by NPU DRAM. Once a prefix is evicted, a later request has to recompute it. SDK 2026.4 adds an experimental hierarchical KV cache that extends reuse across three tiers:

• L1, NPU DRAM: the active, lowest-latency blocks used directly by attention.
• L2, host memory: evicted or backed-up blocks are copied over DMA and can be reloaded instead of recomputed.
• L3, external storage: an optional backend that keeps prefixes beyond local process memory. A shared backend lets other Furiosa-LLM replicas in the cluster reuse them.

Mooncake is the first supported L3 backend, offering content-addressed, cluster-wide prefix storage and asynchronous transfers outside the scheduler's critical path. A backend-agnostic connector interface is meant to allow other storage systems to be added later. The release also replaces separate global-only and hybrid prefix-cache implementations with a unified radix tree implementation.

Two write policies govern L2. Write-through copies a prefix to L2 after it has been reused a set number of times (default: 2), which suits systems with ample NPU-to-host bandwidth and host memory. Write-back writes to L2 on eviction from L1, reducing duplicate storage but possibly adding eviction latency. Both can be enabled together.

The feature is off by default and its options are hidden from the help output of furiosa-llm serve while policies and defaults are tuned. L2 is enabled with a host-memory budget, for example --experimental-kv-offload-host-memory-gb 64. L3 requires a compatible Mooncake deployment and shared-library installation.

Vision-language models and wider model coverage

Vision-language support, introduced in SDK 2026.3, now runs end to end on the standard FCFS scheduler. FuriosaAI says this puts text-only and image-and-text requests under the same scheduling, prefix-cache and data-parallel infrastructure rather than a separate proof-of-concept loop. Three optimizations apply: batched vision encoding across requests, overlapped host-side image preparation with NPU text generation, and reuse of cached image embeddings across conversation turns and data-parallel workers.

The model list also grows. Newly covered or expanded entries include:

• Gemma 4 (31B vision-language) and EXAONE 4.5 (33B vision-language)
• Qwen3-VL 2B and 4B Instruct/Thinking models
• Qwen3 Embedding and Reranker in 0.6B and 4B sizes
• Mistral NeMo 12B (Mistral-Nemo-Instruct-2407-FP8)
• Dynamically quantized FP8 variants of Llama 3.1 and Llama 3.3
• BGE-M3, BGE Reranker v2 M3, E5-Mistral-7B-Instruct and Harrier OSS v1 (0.6B)

FuriosaAI's summary also lists experimental MTP speculative decoding for EXAONE 4.5.

What it means for RNGD users

The release completes the move to Furiosa Executable Bundles (FXB). According to FuriosaAI, published models now ship with precompiled kernels and start without compiling on the serving host.

Teams upgrading from 2026.3 are advised to read the Breaking Changes and Deprecations section and the upgrade guide before moving. For enterprises running RNGD, the main practical effects described by the company are better efficiency on large MoE models, faster startup, and a broader set of supported generation, multimodal and retrieval models. The hierarchical cache is experimental, so production use of it will depend on further tuning by FuriosaAI.