Back to Home
MarkTechPost··Papers & Tech

Google Releases TPU Sync: Open-Source KV Cache Transfer for TPUs

中文摘要

谷歌发布TPU Sync开源库,通过优化内存与网络间的KV缓存传输,显著降低了大模型(235B-480B)的传输延迟,提升了TPU集群的整体性能。

English Summary

Google released TPU Sync, an open-source library optimizing KV cache and weight transfers. It reduces latency by over 90% for large MoE models, significantly improving TPU efficiency and performance.

Original Excerpt

Google released TPU Sync, an open-source C++ library that moves KV caches and weights across TPU HBM, host DRAM, storage and network. Google reports over 90% lower KV transfer latency on 235B to 480B MoE models.