Back to Home
量子位··Domestic Sources

KV Cache砍掉99%,单卡吞吐飙6倍,港科CMU联手岩山科技开源新模型

中文摘要

港科CMU联手岩山科技开源新模型,KV Cache砍99%,单卡吞吐飙6倍,实现512路长推理并发。

English Summary

HKUST, CMU, & Yan Shan Tech's new open-source model slashes KV Cache by 99%. It boosts single-card throughput 6x, enabling 512 concurrent long inferences per card.

Original Excerpt

一张卡,512路长推理同时跑