Joint Knowledge Distillation and Local Differential Privacy for Communication-Efficient Federated Learning in Heterogeneous Systems

Abstract

Federated Learning (FL) has emerged as a powerful approach to facilitate the construction of centralized models without compromising the data privacy of multiple participants. However, conventional FL methodologies do not address system heterogeneity where each participant needs to independently design its own model, a prevalent requirement in Internet of Things (IoT) applications due to the heterogeneous nature of tasks and data. Knowledge Distillation-based FL algorithms tackle this limitation by exchanging soft labels instead of model weights, thus giving each client the ability to independently design its local model architecture. While FL is inherently private, studies have indicated that exploiting gradients for a few iterations can reveal sensitive training data. To protect against privacy attacks, FL algorithms employ Differential Privacy (DP) to guarantee privacy protection, which can be applied using Local Differential Privacy (LDP). In this paper, we elaborate on preserving clients' training data privacy in KD (Knowledge Distillation)-based FL using DP, providing both privacy and flexibility. We provide theoretical analysis to extend the privacy guarantee to exchanged updates. Experimental analysis is performed utilizing Human Activity Recognition (HAR) datasets with different modalities. The results obtained demonstrate the capacity of KD-based FL to maintain a robust utility-privacy balance. Furthermore, for the same DP protection level, the utility of models trained on images was severely reduced across all FL algorithms. This suggests that the modality and complexity of a dataset are important factors for shaping the utility-privacy tradeoff of DP.

Topics & Concepts