MSCCLang: Microsoft Collective Communication Language
Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, Yifan Xiong
Abstract
Machine learning models with millions or billions of parameters are increasingly trained and served on large multi-GPU systems. As models grow in size and execute on more GPUs, collective communication becomes a bottleneck. Custom collective algorithms optimized for both particular network topologies and application-specific communication patterns can alleviate this bottleneck and help these applications scale. However, implementing correct and efficient custom algorithms is challenging.
Topics & Concepts
BottleneckComputer scienceNetwork topologyDistributed computingScale (ratio)Parallel computingComputer networkEmbedded systemPhysicsQuantum mechanicsParallel Computing and Optimization TechniquesDistributed systems and fault toleranceLogic, programming, and type systems