FedRSCLIP: Federated learning for remote sensing scene classification using vision-language models

Abstract

Remote sensing image classification is essential for various applications, including agricultural monitoring, urban planning, and land use classification. However, remote sensing data are often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. Nevertheless, current vision-language models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates as they would incur substantial communication costs. In this article, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically, Contrastive Language-Image Pre-training (CLIP). FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing prompt learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism (DPM), comprising shared prompts for global knowledge sharing and private prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the dual-prompt alignment constraint (DPAC), which balances global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the cross-modal feature alignment constraint (CMFAC) to align multimodal features between text and image prompts.

Topics & Concepts