Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
中文摘要
Buoyant发布Kinference,Kubernetes上的vLLM推理控制平面,提升效率。
English Summary
Buoyant releases Kinference, an open-source inference control plane for vLLM on Kubernetes, improving efficiency.
Original Excerpt
Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark.