Global and Local Prompts Cooperation via Optimal Transport for Federated Learning

Abstract

Prompt learning in pretrained visual-language mod-els has shown remarkable flexibility across various down-stream tasks. Leveraging its inherent lightweight nature, re-cent research attempted to integrate the powerful pretrained models into federated learning frameworks to simultane-ously reduce communication costs and promote local training on insufficient data. Despite these efforts, current fed-erated prompt learning methods lack specialized designs to systematically address severe data heterogeneities, e.g., data distribution with both label and feature shifts involved. To address this challenge, we present Federated Prompts Cooperation via Optimal Transport (FedOTP), which intro-duces efficient collaborative prompt learning strategies to capture diverse category traits on a per-client basis. Specif-ically, for each client, we learn a global prompt to extract consensus knowledge among clients, and a local prompt to capture client-specific category characteristics. Unbal-anced Optimal Transport is then employed to align local vi-sual features with these prompts, striking a balance between global consensus and local personalization. By relaxing one of the equality constraints, FedOTP enables prompts to focus solely on the core regions of image patches. Exten-sive experiments on datasets with various types of hetero-geneities have demonstrated that our FedOTP outperforms the state-of-the-art methods.

Topics & Concepts