NVIDIA Proved 4-Bit Training Works at Real Scale (Not Just Inference)
中文摘要
NVIDIA证明4位训练可大规模实现，性能接近FP8。
English Summary
NVIDIA proves 4-bit training is viable at scale, nearing FP8 performance.
Original Excerpt
A 12-billion-parameter Mamba-Transformer, trained on 10 trillion tokens, entirely in 4-bit floating point, matched an FP8 baseline almost… Continue reading on AIGuys »