ESync: Accelerating Intra-Domain Federated Learning in Heterogeneous Data Centers
Zonghang Li, Huaman Zhou, Tianyao Zhou, Hongfang Yu, Zenglin Xu, Gang Sun
Abstract
Federated Learning (FL) serves privacy-preserving collaborative learning among multiple isolated parties, while retaining their privacy data locally. Cross-device and cross-silo FL have achieved great success in cross-domain applications, in which the scarce communication resource is the primary bottleneck. Driven by the need to combine heterogeneous machines from different parties to build a shared data center, we found <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">intra-domain FL</i> , a new type of FL in which isolated parties collaborate in the shared data center, and strong computational heterogeneity becomes the primary bottleneck. To mitigate the training inefficiency caused by stragglers, this article proposes an efficient synchronization algorithm <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ESync</i> , which allows parties to train different iterations locally under the coordination of a novel scheduler <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">State Server</i> . We give the boundaries of weight divergence and optimality gap of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ESync</i> , and analyze the trade-off between convergence accuracy and communication efficiency. Extensive experiments are conducted to compare <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ESync</i> with SSGD, ASGD, DC-ASGD, FedAvg, FedAsync, TiFL, and FedDrop under strong computational heterogeneity. Numerical results show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ESync</i> achieves great speed up without loss of accuracy, and therefore demonstrate the effectiveness of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ESync</i> in both training efficiency and converged accuracy.