Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation
G. H. Liu, Youshan Miao, Zhiqi Lin, Xiaoxiang Shi, Saeed Maleki, Fan Yang, Yungang Bao, Sa Wang
Abstract
Many parallel mechanisms, including data parallelism, tensor parallelism, and pipeline parallelism, have been proposed and combined together to support training increasingly large deep neural networks (DNN) on massive GPU devices. Given a DNN model and GPU cluster, finding the optimal configuration by combining these parallelism mechanisms is an NP-hard problem. Widely adopted mathematical programming approaches have been proposed to search in a configuration subspace, but they are still too costly when scaling to large models over numerous devices.
Topics & Concepts
Computer scienceBottleneckParallelism (grammar)Parallel computingPipeline (software)Data parallelismProgramming paradigmScalingArtificial intelligenceEmbedded systemMathematicsProgramming languageGeometryAdvanced Neural Network ApplicationsStochastic Gradient Optimization TechniquesParallel Computing and Optimization Techniques