GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, Yida Wang
Abstract
Large deep learning models have recently garnered substantial attention from both academia and industry. Nonetheless, frequent failures are observed during large model training due to large-scale resources involved and extended training time. Existing solutions have significant failure recovery costs due to the severe restriction imposed by the bandwidth of remote storage in which they store checkpoints.
Topics & Concepts
Computer scienceTraining (meteorology)Bandwidth (computing)Scale (ratio)Distributed computingDistributed data storeComputer networkPhysicsQuantum mechanicsMeteorologyAdvanced Neural Network ApplicationsPrivacy-Preserving Technologies in DataStochastic Gradient Optimization Techniques