Back to Home
MarkTechPost··Papers & Tech

Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

中文摘要

Buoyant发布Kinference,Kubernetes上的vLLM推理控制平面,提升效率。

English Summary

Buoyant releases Kinference, an open-source inference control plane for vLLM on Kubernetes, improving efficiency.

Original Excerpt

Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark.