In-Network Aggregation with Transport Transparency for Distributed Training
Shuo Liu, Qiaoling Wang, Junyi Zhang, Wenfei Wu, Qinliang Lin, Yao Liu, Meng Xu, Marco Canini, Ray C. C. Cheung, Jianfei He
Abstract
Recent In-Network Aggregation (INA) solutions offload the all-reduce operation onto network switches to accelerate and scale distributed training (DT). On end hosts, these solutions build custom network stacks to replace the transport layer. The INA-oriented network stack cannot take advantage of the state-of-the-art performant transport layer implementation, and also causes complexity in system development and operation.
Topics & Concepts
Computer scienceTransparency (behavior)Distributed computingTransport layerStack (abstract data type)Transport networkLayer (electronics)Computer networkOperating systemComputer securityNanotechnologyMaterials scienceSoftware-Defined Networks and 5GIoT and Edge/Fog ComputingEnergy Efficient Wireless Sensor Networks