Synthesizing optimal collective algorithms
Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi
Abstract
Collective communication algorithms are an important component of distributed computation. Indeed, in the case of deep-learning, collective communication is the Amdahl's bottleneck of data-parallel training.
Topics & Concepts
Computer scienceBottleneckComponent (thermodynamics)AlgorithmTheoretical computer scienceKey (lock)Efficient algorithmDistributed computingDistributed algorithmAlgorithm designStochastic Gradient Optimization TechniquesFerroelectric and Negative Capacitance DevicesAdvanced Memory and Neural Computing