Balancing Privacy and Accuracy Using Significant Gradient Protection in Federated Learning
Benteng Zhang, Yingchi Mao, Xiaoming He, Huawei Huang, Jie Wu
Abstract
Previous state-of-the-art studies have demonstrated that adversaries can access sensitive user data by membership inference attacks (MIAs) in Federated Learning (FL). Introducing differential privacy (DP) into the FL framework is an effective way to enhance the privacy of FL. Nevertheless, in differentially private federated learning (DP-FL), local gradients become excessively sparse in certain training rounds. Especially when training with low privacy budgets, there is a risk of introducing excessive noise into clients’ gradients. This issue can lead to a significant degradation in the accuracy of the global model. Thus, how to balance the user's privacy and global model accuracy becomes a challenge in DP-FL. To this end, we propose an approach, known as differential privacy federated aggregation, based on significant gradient protection (DP-FedASGP). DP-FedASGP can mitigate excessive noises by protecting significant gradients and accelerate the convergence of the global model by calculating dynamic aggregation weights for gradients. Experimental results show that DP-FedASGP achieves comparable privacy protection effects to DP-FedAvg and cpSGD (communication-private SGD based on gradient quantization) but outperforms DP-FedSNLC (sparse noise based on clipping losses and privacy budget costs) and FedSMP (sparsified model perturbation). Furthermore, the average global test accuracy of DP-FedASGP across four datasets and three models is about <inline-formula><tex-math notation="LaTeX">$2.62$</tex-math></inline-formula>%, <inline-formula><tex-math notation="LaTeX">$4.71$</tex-math></inline-formula>%, <inline-formula><tex-math notation="LaTeX">$0.45$</tex-math></inline-formula>%, and <inline-formula><tex-math notation="LaTeX">$0.19$</tex-math></inline-formula>% higher than the above methods, respectively. These improvements indicate that DP-FedASGP is a promising approach for balancing the privacy and accuracy of DP-FL.