返回首页
MarkTechPost··论文与技术

Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

中文摘要

Buoyant发布Kinference,Kubernetes上的vLLM推理控制平面,提升效率。

English Summary

Buoyant releases Kinference, an open-source inference control plane for vLLM on Kubernetes, improving efficiency.

原文节选

Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark.