OCP静默数据损坏工作组更新.pdf

1、Harish Dixit,MetaCyril Meurillon,NVIDIASilent Data Corruption UpdatesSDC Challenges in AISilent Data Corruption UpdatesHarish Dixit,MetaCyril Meurillon,NVIDIAHardware MAnagementSilent Data Corruption(Refresher)Defectsin silicon2+2=5Silent Errors in Compute Units Hard to detect Undetected formonths/y

2、earsSignificant impact to servicesWhat makes SDCs different?Faulty DeviceTypical Fault ManagementECCs,Logs,Counters,RAS Features SDCs in AI workloadsmay cause numeric explosions,e.g.NaN or subtly undermine model accuracyGot NaN?SDC:Hardware faults that go undetected,subtly undermines AI model accura

3、cy and trustworthiness.Growing Urgency:AI/ML at HPC scale(billions of parameters,thousands of nodes)amplifies SDC impact.Difficulty in Correlation:Hard to link low-level hardware errors to high-level AI performance.Insidious Nature:Produces incorrect outputs without triggering alarms.Impact on Trust

4、:Compromises the integrity&reliability of AI systems,especially in critical applications.SDC Challenges in AIDrive solutions and best practices that prevent and detect SDCs.Create awareness about SDC challenges across the computing community.Partner&engage with the academic community to actively add

5、ress growing SDC challenges.OCP Server Resilience,SDC Working GroupSpecification 1.0:released in 2024Training Implications:NaN Propagation:SDCs can lead to Not-a-Number values,propagating across clusters and causing halts and significant debug time.Corrupted Gradients:Subtle corruptions can cause tr

6、aining to stall or diverge.Computational Inefficiency:Wastes weeks/months of valuable computational resourcesSDC Impact on AI TrainingInference Implications:Incorrect Results:Corrupted devices yield inaccurate predictions,directly affecting critical decisions.Costly Triage:Identifying and quarantini