RAT - Resilient Allreduce Tree for Distributed Machine Learning
Xinchen Wan, Hong Zhang, Hao Wang, Shuihai Hu, Junxue Zhang, Kai Chen
Abstract
Parameter/gradient exchange plays an important role in large-scale distributed machine learning (DML). However, prior solutions such as parameter server (PS) or ring-allreduce (Ring) fall short since they are not resilient to issues or uncertainties like oversubscription, congestion or failures that may occur in datacenter networks (DCN).
Topics & Concepts
Computer scienceDistributed computingServerScale (ratio)Ring (chemistry)Computer networkTree (set theory)Artificial intelligenceOperating systemMathematicsQuantum mechanicsOrganic chemistryPhysicsChemistryMathematical analysisCloud Computing and Resource ManagementData Stream Mining TechniquesScientific Computing and Data Management