arXiv:2503.14938cs.CV2025-03被引 2

用最优传输优化跨模态表示,提升少样本遥感分类效果

Optimal Transport Adapter Tuning for Bridging Modality Gaps in Few-Shot Remote Sensing Scene Classification

  • 通过最优传输构建理想表征空间,融合视觉与文本信息
  • 在基准数据集上显著提升分类准确率,超越现有方法
  • 适合遥感图像少样本学习与多模态融合研究者使用

少样本遥感场景分类(FS-RSSC)面临标注样本极少的挑战。现有方法多聚焦单模态特征学习,忽视多模态表示优化潜力。为此,我们提出一种基于最优传输理论的适配器调优框架(OTAT),旨在通过最优传输构建理想的柏拉图表征空间,实现丰富视觉信息与稀疏文本线索的有效融合,促进跨模态信息互补与传递。核心是交叉模态注意力机制的最优传输适配器(OTA),可增强文本表征并促进后续信息交互。将网络优化转化为最优传输优化问题,建立模态间平衡的信息交换路径。此外,引入样本级熵感知加权(EAW)损失,结合难度加权相似度与基于熵的正则化,细化控制最优传输过程,提升其可解性与稳定性。大量实验表明,该框架在多个基准数据集上实现最先进的少样本遥感分类性能,显著提升模型表现与泛化能力。

原文摘要 · Abstract (English)

Few-Shot Remote Sensing Scene Classification (FS-RSSC) presents the challenge of classifying remote sensing images with limited labeled samples. Existing methods typically emphasize single-modal feature learning, neglecting the potential benefits of optimizing multi-modal representations. To address this limitation, we propose a novel Optimal Transport Adapter Tuning (OTAT) framework aimed at constructing an ideal Platonic representational space through optimal transport (OT) theory. This framework seeks to harmonize rich visual information with less dense textual cues, enabling effective cross-modal information transfer and complementarity. Central to this approach is the Optimal Transport Adapter (OTA), which employs a cross-modal attention mechanism to enrich textual representations and facilitate subsequent better information interaction. By transforming the network optimization into an OT optimization problem, OTA establishes efficient pathways for balanced information exchange between modalities. Moreover, we introduce a sample-level Entropy-Aware Weighted (EAW) loss, which combines difficulty-weighted similarity scores with entropy-based regularization. This loss function provides finer control over the OT optimization process, enhancing its solvability and stability. Our framework offers a scalable and efficient solution for advancing multimodal learning in remote sensing applications. Extensive experiments on benchmark datasets demonstrate that OTAT achieves state-of-the-art performance in FS-RSSC, significantly improving the model performance and generalization.

遥感分类少样本学习跨模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。