FlexReduce: Flexible All-reduce for Distributed Deep Learning on Asymmetric Network Topology

Jinho Lee, Inseok Hwang, Soham Shah, Minsik Cho

202021 citationsDOI

Abstract

We propose FlexReduce, an efficient and flexible all-reduce algorithm for distributed deep learning under irregular network hierarchies. With ever-growing deep neural networks, distributed learning over multiple nodes is becoming imperative for expedited training. There are several approaches leveraging the symmetric network structure to optimize the performance over different hierarchy levels of the network. However, the assumption of symmetric network does not always hold, especially in shared cloud environments. By allocating an uneven portion of gradients to each learner (GPU), FlexReduce outperforms conventional algorithms on asymmetric network structures, and still performs even or better on symmetric networks.

Topics & Concepts

Computer scienceDistributed computingDeep learningNetwork topologyHierarchical network modelHierarchyCloud computingDistributed learningArtificial intelligenceArtificial neural networkTopology (electrical circuits)Theoretical computer scienceComputer networkEngineeringOperating systemMarket economyEconomicsElectrical engineeringPedagogyPsychologyAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and ELM