Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

Amazon SageMaker HyperPod has introduced three new checks within PyTorch DistributedDataParallel training to detect and recover from silent data corruption (SDC) on GPUs. These checks identify non-finite gradients, replay micro-batches for audit, and compare weight fingerprints to isolate and replace faulty nodes automatically. The approach was validated on a cluster with NVIDIA H100 GPUs using software-injected faults.