Quasi-global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
Tao Lin, Sai Praneeth Karimireddy, Sebastian U. Stich, Martin Jaggi
Abstract
Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge and may severely deteriorate the generalization performance.
Topics & Concepts
Computer scienceGeneralizationNetwork topologyMachine learningKey (lock)Artificial intelligenceDeep learningCode (set theory)Data miningDistributed computingComputer securitySet (abstract data type)Mathematical analysisOperating systemMathematicsProgramming languagePrivacy-Preserving Technologies in DataStochastic Gradient Optimization TechniquesHuman Mobility and Location-Based Analysis