Kinference: Buoyant Releases Open Source Inference Control Plane for vLLM
Buoyant has released Kinference, an open source control plane for building inference platforms on vLLM and Kubernetes. According to the company, its KV-cache-aware load balancing cut median time-to-first-token by 95% against standard Kubernetes load balancing on one workload type.
- Buoyant released Kinference, an open source control plane for vLLM and Kubernetes, under Apache v2.
- It provides token rate limiting, tenant usage and cost tracking, load balancing and inference policies.
- According to Buoyant, KV-cache-aware routing cut median TTFT by 95% and median end-to-end latency by 45% on one workload type.
- In the capacity benchmark, goodput was 7.4× on shared-prefix and about 3.5× on multiturn workloads.
- YAML policies can route by complexity, burst to cloud providers or keep PII-containing prompts local.
What Kinference does
Kinference was created at Buoyant after the company watched customers struggle to run open weight models in their own environments. It is designed as the control plane layer of an inference platform, while vLLM handles the underlying inference.
The feature set covers what Buoyant describes as necessary for building a platform in practice: token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies.
Kinference can be paired with an existing stack. Teams point their inference clients to its HTTP endpoint and send OpenAI or Anthropic-compatible traffic there. According to Buoyant, it serves a heterogeneous, evolving set of models running on a heterogeneous, evolving pool of GPUs, for a similarly varied set of clients.
Cache-aware load balancing and benchmark results
Kinference's load balancing is specific to inference. It takes into account signals such as GPU queue sizes, historical performance and the contents of the KV cache on the GPU. With this KV-cache awareness, it can send a request to the vLLM worker that will do the least work for that specific request.
Buoyant points to its public benchmarks. Compared with standard Kubernetes load balancing, Kinference cut median time-to-first-token (TTFT) by 95% and median end-to-end latency by 45% on one workload type, according to the company.
In a separate capacity benchmark, which measured how many requests per minute each endpoint could serve within a latency SLO, Kinference delivered 7.4× the goodput on a shared-prefix workload and about 3.5× on a multiturn workload. It reached the chosen SLO at concurrency 32 (the highest measured) on the shared-prefix workload and at concurrency 8 on the multiturn workload. The kube-proxy approach did not reach the SLO at any concurrency level.
Inference policies
The central idea in Kinference is the inference policy, which lets platform owners balance tradeoffs across usage, performance and priorities. Buoyant says customers asked how to let users request broader classes of models, such as "any Qwen 3.8 model" or "whatever is good for coding", instead of specific checkpoints. They also asked how to redirect easy requests to cheaper models without sending a whole prompt history to a cold GPU and taking a large latency hit.
Instead of building one-off features, Buoyant built a policy layer, defined in YAML. The announcement gives three examples:
- Semantic routing: classify prompts by complexity and route simpler ones to cheaper models, unless cache usage makes an expensive model faster.
- Cloud bursting: send excess requests to cloud provider endpoints when queue depth or cache pressure shows the GPUs are saturated.
- PII handling: classify prompts that contain PII so they stay on local infrastructure, with requests rejected (fail closed) if no local option is available.
Policies can be based on tenant, requested model, GPU pool state, semantic classification and other factors. They can go through human or agentic review, be committed to git, be linted and vetted, and be applied or rolled back independently.
Metrics, installation and availability
Kinference also exposes metrics for monitoring, alerting and SLOs. These include token rates in and out, histograms of input and output token sizes, TTFT, ITL and end-to-end latency histograms, cached versus uncached prompt tokens for cost estimation, model usage and GPU saturation. Almost every metric is also available per tenant.
Installation uses Helm and the Gateway API CRDs. The steps are to install the Gateway API CRDs, then the Kinference gateway, then Kinference and its CRDs, and finally to verify that the pods are running. Buoyant says the steps will evolve and directs users to the documentation at kinference.ai/docs.
Kinference is fully open source under Apache v2 and available on GitHub. Buoyant welcomes feedback, issues and benchmark reports from other deployments.
What it means for platform teams
For teams running open weight models on their own GPUs, Kinference offers a ready-made layer for routing, usage tracking and policy control on top of vLLM. The benchmark figures come from Buoyant's own tests, so results in other environments may differ, which is why the company asks for reports from other deployments.
