用视觉语言模型做遥感图像分类,解决数据分散与通信成本高的问题。
FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models
- 通过提示学习只更新少量参数,降低联邦学习通信开销。
- 双提示机制提升全局共享与本地适应的平衡,准确率达92.3%。
- 适合遥感、医疗等隐私敏感领域的分布式模型训练。
遥感数据通常分布在多个机构中,由于隐私和数据共享限制,难以在集中式框架下使用大规模数据集进行训练。联邦学习提供了一种解决方案,可在不集中数据的情况下实现跨数据源的协作训练。然而,当前的视觉语言模型(VLMs)通常包含数十亿参数,基于模型参数更新的传统联邦学习方法会带来巨大通信开销。本文提出首个基于视觉语言模型CLIP的遥感图像分类联邦学习框架FedRSCLIP。该框架通过提示学习仅优化少量可调参数,以应对数据异构性和大规模模型传输挑战。引入双提示机制,包括用于全局知识共享的共享提示和用于客户端适配的私有提示,并设计双提示对齐约束以平衡全局一致性与局部适应性。此外,为增强跨模态表示学习,提出跨模态特征对齐约束,对齐文本与图像提示间的多模态特征。为验证模型有效性,我们基于三个现有遥感图像分类数据集构建了联邦遥感图像分类数据集Fed-RSIC,模拟多种联邦学习配置。实验结果表明,FedRSCLIP在遥感图像分类任务中表现优异,具有显著优势。
原文摘要 · Abstract (English)
Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。