Back to Home
AWS ML Blog··Official Labs

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

中文摘要

Amazon SageMaker HyperPod 在 PyTorch DDP 循环中增加三项检查,用于检测、定位并自动修复 GPU 静默数据损坏,以保护分布式训练。

English Summary

Amazon SageMaker HyperPod adds three PyTorch DDP loop checks to detect, localize, and automatically recover from silent GPU data corruption during distributed training.

Original Excerpt

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.