Baseten's AI-built inference engine beats vLLM by up to 90% on Qwen decode

From a research paper to a week-long agent run

The experiment began in an internal Slack channel at Baseten, where a paper called MetaInfer: A Knowledge Only LLM Inference Engine Generator SKILL Toolbox caught the attention of the post's author. The authors describe a skills-only framework for building custom inference engines from scratch. "Skills-only" means no model post-training and no specialized harnesses. They claim their results beat open solutions such as vLLM on metrics like time per output token (TPOT).

The author says he was skeptical but tried the provided repository on a popular model. He gave Claude Code with Fable 5 access to the paper, the MetaInfer repository and an NVIDIA B200 workstation over SSH. He also supplied the NVFP4 quantized weights for Qwen-3.6-35B-A3B and the original full-precision weights to serve as an accuracy oracle. The agent could also deploy engine candidates to Baseten and run AIPerf profiles against the deployed endpoint.

The goal he set was to beat vLLM by 20% on all performance metrics without accuracy loss versus the NVFP4 baseline. According to Baseten, Claude worked mostly autonomously for roughly 1 week, with occasional human steering to avoid over-fixating on one traffic shape and to keep it focused on production performance rather than isolated engine benchmarks.

The results against vLLM

The agent consumed roughly 1.7 billion tokens, overwhelmingly cached input tokens, and about 200 B200 hours. It reached parity with vLLM within the first few days, but the author kept it running. In the end, VibeQwen outperformed vLLM across all tested traffic patterns, according to Baseten.

• Single-stream decode: 1,792 tokens per second on speculator-friendly text (repetitive and structured), versus 943 TPS on a well-tuned vLLM deployment, a 90% improvement.
• First token latency: 12ms on the tested corpus versus 28ms with vLLM, a 2.3x improvement.
• Throughput at concurrency of 32 (single replica): 10,307 output TPS versus 6,030 for vLLM, a 71% gain.

Baseten says this opens up ultra-low-latency use cases for the model that had not been practical before.

Why inference suits automated optimization

Baseten points to several converging trends. Models are getting better at long-horizon tasks, with the post citing Anthropic's Mythos 5 and Fable 5. Agent coordination features are also being built into popular harnesses, including "Goals" in Claude Code and Codex, which keep models working toward measurable targets over long periods. Meanwhile, demand for inference keeps growing, which puts pressure on cost and performance.

The author argues inference performance is an ideal domain for this approach. Metrics such as TTFT, TPOT, throughput per chip and required memory are quantifiable, and correctness can be checked numerically against full-precision reference implementations. The search space is also large, from CUDA kernel authoring to serving-layer techniques like dynamic batching, and there is a big gap between current state-of-the-art performance and the theoretical maximum.

Constraints matter, and Baseten changed the setup

The post warns that agents will try to game a goal. Without accuracy-preserving constraints, a model could converge on a loop that yields the letter "a", which would score extremely well on TTFT and TPOT but be useless. Designing the game so that the agent's success matches real needs is the engineer's job, the author writes.

The premise of MetaInfer is that a highly specialized engine for one model, accelerator and workload can beat general engines like vLLM, SGLang and TensorRT-LLM, which trade peak optimization for breadth and ease of use. Baseten departed from the paper's experimental purity: agents could consult open solutions for reference, including pre-optimized kernels from vLLM and TensorRT-LLM, and wrote their own kernels only when needed. The scope also covered the full serving stack, measured on deployed multi-replica services behind a load balancer using AIPerf.

On accuracy, the author allowed subtly different numerical outputs from the reference NVFP4 implementation, as long as accuracy versus the BF16 baseline was at least as good. Each run yields two deliverables: the engine and reusable extensions to the knowledge base for later runs.

What it means

For teams serving a fixed model on fixed hardware, Baseten's result suggests that agent-generated, narrowly specialized engines can beat general-purpose ones when the goal and accuracy constraints are well defined. The figures come from Baseten's own tests on one model and one GPU type, so they should be read as a single case study rather than a general benchmark.