Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning: Examining Distributed and Centralized Stochastic Gradient Descent
Shi Pu, Alex Olshevsky, Ioannis Ch. Paschalidis
Abstract
We provide a discussion of several recent results which, in certain scenarios, are able to overcome a barrier in distributed stochastic optimization for machine learning (ML). Our focus is the so-called asymptotic network independence property, which is achieved whenever a distributed method executed over a network of n nodes asymptotically converges to the optimal solution at a comparable rate to a centralized method with the same computational power as the entire network. We explain this property through an example involving the training of ML models and sketch a short mathematical analysis for comparing the performance of distributed stochastic gradient descent (DSGD) with centralized SGD.
Topics & Concepts
Stochastic gradient descentComputer scienceIndependence (probability theory)Gradient descentStochastic optimizationMathematical optimizationDescent (aeronautics)Artificial intelligenceDistributed computingArtificial neural networkMathematicsEngineeringStatisticsAerospace engineeringStochastic Gradient Optimization TechniquesDistributed Control Multi-Agent SystemsSparse and Compressive Sensing Techniques