会议3_利用现代 GPU 集群基于 GPU 的高效压缩方案加速 MPI ALLREDUCE 通信.pdf

1、ACCELERATING MPI ALLREDUCECOMMUNICATION WITH EFFICIENT GPU-BASED COMPRESSION SCHEMES ON MODERN GPU CLUSTERS2024 OFA Virtual WorkshopHari Subramoni and Qinghua ZhouThe Ohio State UniversityEmail:subramoni.1,Zhou.2595osu.eduPRESENTATION OVERVIEWIntroduction&MotivationDesign ApproachesRing AllReduce wi

2、th Collective-level Online CompressionRecursive-Doubling AllReduce with Collective-level Online CompressionPerformance EvaluationBenchmark-level evaluationApplication-level evaluationConclusion and Future Plan2 OpenFabrics AllianceINTRODUCTION AND MOTIVATIONAllReduce is a communication collective op

3、eration that is commonly used in HPC applications as well as distributed DL training.Existing AllReduce algorithms for transferring large GPU data still suffer from poor performance due to the limited interconnect bandwidth of networksNaive point-to-point compression for each data transmission may i

4、ntroduce redundant compression/decompression operations and hinder non-blocking send/receive operationsHow to co-design and optimize the GPU-based compression at the collective-level along with the communication patterns of advanced AllReduce algorithms?We propose two design approaches along with th

5、ese directions.Ring AllReduce with Collective-level Online CompressionRecursive-Doubling AllReduce with Collective-level Online Compression3 OpenFabrics AlliancePRESENTATION OVERVIEWIntroduction&MotivationDesign ApproachesRing AllReduce with Collective-level Online CompressionRecursive-Doubling AllR

6、educe with Collective-level Online CompressionPerformance EvaluationBenchmark-level evaluationApplication-level evaluationConclusion and Future Plan4 OpenFabrics AllianceDESIGN APPROACHESRing and Recursive-Doubling MPI_AllReducewith Collective-level Online CompressionCompression can reduce the data