RDMA over Ethernet for Distributed Training at Meta Scale
Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, Shuqiang Zhang, Mikel Jimenez Fernandez, Shashidhar Gandham, Hongyi Zeng
Abstract
The rapid growth in both computational density and scale in AI models in recent years motivates the construction of an efficient and reliable dedicated network infrastructure. This paper presents the design, implementation, and operation of Meta's Remote Direct Memory Access over Converged Ethernet (RoCE) networks for distributed AI training.
Topics & Concepts
Remote direct memory accessEthernetComputer scienceScale (ratio)Computer networkOperating systemGeographyCartographyDistributed and Parallel Computing SystemsPeer-to-Peer Network TechnologiesParallel Computing and Optimization Techniques