KV Cache砍掉99%,单卡吞吐飙6倍,港科CMU联手岩山科技开源新模型
中文摘要
港科CMU联手岩山科技开源新模型,KV Cache砍99%,单卡吞吐飙6倍,实现512路长推理并发。
English Summary
HKUST, CMU, & Yan Shan Tech's new open-source model slashes KV Cache by 99%. It boosts single-card throughput 6x, enabling 512 concurrent long inferences per card.
原文节选
一张卡,512路长推理同时跑