返回首页
AWS ML Blog··官方实验室

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

中文摘要

Amazon SageMaker HyperPod 在 PyTorch DDP 循环中增加三项检查,用于检测、定位并自动修复 GPU 静默数据损坏,以保护分布式训练。

English Summary

Amazon SageMaker HyperPod adds three PyTorch DDP loop checks to detect, localize, and automatically recover from silent GPU data corruption during distributed training.

原文节选

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.