Large-Scale Decentralized Asynchronous Federated Edge Learning with Device Heterogeneity
Xuan Liang, Jianhua Tang, Tony Q. S. Quek
Abstract
In conventional federated learning (FL), there exists a single point of failure in the central server. Thus the studies about decentralized federated learning (DFL) paradigm have become popular recently. In DFL, some clients with poor computation capacity may take a long time to train local models, therefore, the convergence speed of the global model is usually slow in existing synchronous algorithms. In this work, we consider a large-scale system with device heterogeneity. To reduce training time and fully utilize edge node computation capacity, we propose an asynchronous algorithm in a novel multi-cluster decentralized federated edge learning (MD-FEEL) framework, where there are many clusters and each cluster consists of some clients. Our proposed asynchronous MD-FEEL contains four steps, i.e., local stochastic gradient descent (SGD) update, gradient consensus, intra-cluster model aggregation and inter-cluster model aggregation. To measure the staleness of cluster model, we introduce age of update (AoU) in inter-cluster aggregation stage and theoretically prove the convergence of our proposed algorithm on a non-convex setting. We evaluate our asynchronous MD-FEEL on MNIST and CIFAR-10 datasets and the simulation results show it can aggregate to a global model with better accuracy performance and faster convergence speed than some existing synchronous algorithms.