Bypassing HBM Bottlenecks: Fusing Self-Stabilizing Transformers with Custom Triton Kernels
中文摘要
通过将自稳定Transformer与自定义Triton内核融合，消除中间张量实例化，以节省VRAM并降低大规模LLM训练的梯度方差，突破HBM瓶颈。
English Summary
Fusing Self-Stabilizing Transformers with custom Triton kernels eliminates intermediate tensor materialization, saving VRAM and bounding gradient variance to bypass HBM bottlenecks in large-scale LLM training.
Original Excerpt
How eliminating intermediate tensor materialization saves VRAM and bounds gradient variance during large-scale LLM training. Continue reading on Medium »