Federated Split Learning via Mutual Knowledge Distillation
Linjun Luo, Xinglin Zhang
Abstract
Federated learning (FL) and split learning (SL) can coordinate multiple clients (e.g., end devices in mobile/IoT networks) to collaboratively train deep learning models without exposing clients' raw data. Generally, FL converges faster than SL, while SL requires less client-side resource than FL. To excavate their complementary advantages, recent works mostly apply FL to improve SL. These one-way synergy approaches hence face the same limitation as SL, i.e., the clients rely on an online server or download the entire large model to make inferences. Furthermore, they cannot handle the requirements on client-side model personalization. To address these, we propose a novel Federated Split learning framework via Mutual Knowledge Distillation (FSMKD), which intertwines FL with SL in a two-way manner, i.e., we enable FL and SL to boost each other's performance. Specifically, we design a two-body structure including the head:personalized-local-body:tail network as the local model and the head:shared-server-body:tail network as the global model. Thus, FSMKD can support personalized local models and exploit information across heterogeneous learning tasks for the global model training through deep mutual learning. Extensive experiments show that FSMKD outperforms the existing FL-SL synergy frameworks on the server-side and obtains a personalized model that outperforms FedAvg on the client-side.