Is Network the Bottleneck of Distributed Training?
Zhen Zhang, Chaokun Chang, Haibin Lin, Yida Wang, Raman Arora, Xin Jin
Abstract
Recently there has been a surge of research on improving the communication efficiency of distributed training. However, little work has been done to systematically understand whether the network is the bottleneck and to what extent.
Topics & Concepts
BottleneckComputer scienceTraining (meteorology)Distributed computingWork (physics)Computer networkEngineeringEmbedded systemMeteorologyMechanical engineeringPhysicsStochastic Gradient Optimization TechniquesAdvanced Memory and Neural ComputingDistributed Control Multi-Agent Systems