Deep Compression for Efficient and Accelerated Over-the-Air Federated Learning
Fazal Muhammad Ali Khan, Hatem Abou-Zeid, Syed Ali Hassan
Abstract
Over-the-air federated learning (OTA-FL) is a distributed machine learning technique where multiple devices collaboratively train a shared model without sharing their raw data with a central server. The devices exchange model updates concurrently over-the-air and they are aggregated without the need for dedicated wireless resources for each device. A major challenge in OTA-FL is that edge devices are limited in their computation, energy, and communication resources. To address this, we investigate how deep neural network compression techniques can be applied in an OTA-FL system. We propose a compression pipeline comprised of pruning and quantization-aware training that significantly reduces both the computation and communication requirements while maintaining an on-par accuracy to the uncompressed models. We thoroughly investigate the reduction in model size, accuracy, and convergence when pruning and quantization-aware training are applied individually and collectively at multiple pruning and quantization levels. Detailed experiments are conducted on two-different deep learning models and two-different datasets, and under varying signal-to-noise ratios (SNRs) and numbers of clients. We then thoroughly present and discuss the resulting trade-offs and findings. Our results demonstrate that deep compression is very effective in an OTA-FL system and negligible losses in accuracy are possible while maintaining up to 80 percent reductions in model size.