Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters
Zizhao Mo, Huanle Xu, Chengzhong Xu
Abstract
Modern GPU clusters inherently exhibit heterogeneity, encompassing various aspects such as computation and communication. This heterogeneity poses a significant challenge for the elastic scheduling of deep learning workloads. Unfortunately, existing elastic schedulers often overlook the impact of heterogeneity on scaling efficiency, resulting in considerably prolonged job completion times.
Topics & Concepts
Computer scienceScheduling (production processes)ScalingComputationArtificial intelligenceDeep learningDistributed computingMachine learningAlgorithmMathematical optimizationMathematicsGeometryStochastic Gradient Optimization TechniquesAdvanced Neural Network ApplicationsCloud Computing and Resource Management