融合云中光学与SAR数据，低成本提升遥感场景分类准确率
Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Images
- 用知识蒸馏实现异构数据间知识迁移
- 样本级动态平衡多模态信息贡献，缓解模态失衡
- 适合处理含云遥感图像的场景分类任务
在遥感场景分类中，利用预训练的光学模型进行迁移是解决标注数据稀缺的有效方法。然而，云层遮挡导致光学数据信息丢失，显著改变特征分布，影响迁移模型的可靠性与稳定性。现有方法或依赖大量辅助数据进行云去除，或直接使用SAR数据而忽略光学数据中未被遮挡的部分。本文提出一种协同迁移方法，低成本地将无云光学数据训练的源模型迁移到包含云中光学与SAR数据的目标域。框架包含两部分：(1) 基于知识蒸馏的协同迁移策略，实现跨异构数据的知识高效传递；(2) 提出信息调节机制（IRM），通过辅助模型衡量各模态贡献差异，在样本级自动平衡目标模型学习过程中的多模态信息利用。在模拟与真实云数据集上开展迁移实验，结果表明该方法在云覆盖场景下优于其他方案。同时验证了IRM的重要性与局限性，并可视化分析了模型迁移中的模态失衡问题。代码已开源。
原文摘要 · Abstract (English)
In remote sensing scene classification, leveraging the transfer methods with well-trained optical models is an efficient way to overcome label scarcity. However, cloud contamination leads to optical information loss and significant impacts on feature distribution, challenging the reliability and stability of transferred target models. Common solutions include cloud removal for optical data or directly using Synthetic aperture radar (SAR) data in the target domain. However, cloud removal requires substantial auxiliary data for support and pre-training, while directly using SAR disregards the unobstructed portions of optical data. This study presents a scene classification transfer method that synergistically combines multi-modality data, which aims to transfer the source domain model trained on cloudfree optical data to the target domain that includes both cloudy optical and SAR data at low cost. Specifically, the framework incorporates two parts: (1) the collaborative transfer strategy, based on knowledge distillation, enables the efficient prior knowledge transfer across heterogeneous data; (2) the information regulation mechanism (IRM) is proposed to address the modality imbalance issue during transfer. It employs auxiliary models to measure the contribution discrepancy of each modality, and automatically balances the information utilization of modalities during the target model learning process at the sample-level. The transfer experiments were conducted on simulated and real cloud datasets, demonstrating the superior performance of the proposed method compared to other solutions in cloud-covered scenarios. We also verified the importance and limitations of IRM, and further discussed and visualized the modality imbalance problem during the model transfer. Codes are available at https://github.com/wangyuze-csu/ESCCS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。