Personalized Federated Learning for Heterogeneous Edge Device: Self-Knowledge Distillation Approach

Abstract

Federated learning (FL) has become increasingly popular and distributes machine learning models among a large set of resource-constraint edge devices without transferring data to the centralized server. FL seeks to learn a common global model using a set of local models, trained in distributed edge devices, coordinated by a central device. However, a set of slow processing edge devices fail to transmit the locally updated model parameters to the centralized device due to a lack of bandwidth connectivity. Additionally, the data residing across edge devices is statistically diverse, i.e., the distribution of non-IID data. Such communication overhead and the variety of data distribution are two major obstacles to the practical application of FL. Motivated by the challenges, in this research work, we develop a self-knowledge distillation-enabled Personalized Federated Learning framework, namely PerFed-SKD. By enabling edge devices to transfer knowledge from older local models to more recent personalized models, the proposed PerFed-SKD speeds up the process by recalling the historical personalized knowledge for the most recent initialized model. Extensive experiments on two publicly available datasets, i.e., MNIST and EMNIST with various data distribution settings demonstrate the outperformance of the proposed PerFed-SKD over the state-of-the-art methods.

Topics & Concepts