arXiv:2411.15770cs.CV2024-11被引 10

融合光学与雷达遥感图像,提升复杂环境下问答准确率。

Text-Guided Coarse-to-Fine Fusion Network for Robust Remote Sensing Visual Question Answering

  • 用文本引导的渐进式注意力机制聚焦关键区域
  • 自适应融合光学与雷达特征,提升跨模态理解能力
  • 首个大规模光学-雷达遥感问答数据集,覆盖16类问题

遥感视觉问答(RSVQA)受到广泛关注,但现有方法受限于光学传感器成像机制,在云层遮挡和低光照等挑战场景下表现不佳。合成孔径雷达(SAR)具备全天时、全天候成像能力,因此研究光学与SAR图像融合对提升RSVQA性能至关重要。本文提出文本引导的粗到细融合网络(TGFNet),利用问题文本与多源图像间的语义关系,指导特征层面的互补融合。设计文本引导的粗到细注意力精炼(CFAR)模块,通过关键区域路由逐步聚焦从宏观到细节的区域,增强模型对相关区域的关注能力。提出自适应多专家融合(AMEF)模块,动态整合不同专家,实现光学与SAR特征的自适应融合。此外,构建首个大规模光学-SAR RSVQA基准数据集,包含6,008对对齐的光学-SAR图像及1,036,694个标注问答对,涵盖16种多样问题类型,包括复杂关系推理问题。在该数据集上的大量实验表明,TGFNet能有效融合光学与SAR图像的互补信息,显著提升模型在挑战性场景下的性能。数据集已公开:https://github.com/mmic-lcl/。

原文摘要 · Abstract (English)

Remote Sensing Visual Question Answering (RSVQA) has gained significant research interest. However, current RSVQA methods are limited by the imaging mechanisms of optical sensors, particularly under challenging conditions such as cloud-covered and low-light scenarios. Given the all-time and all-weather imaging capabilities of Synthetic Aperture Radar (SAR), it is crucial to investigate the integration of optical-SAR images to improve RSVQA performance. In this work, we propose a Text-guided Coarse-to-Fine Fusion Network (TGFNet), which leverages the semantic relationships between question text and multi-source images to guide the network toward complementary fusion at the feature level. Specifically, we develop a Text-guided Coarse-to-Fine Attention Refinement (CFAR) module to focus on key areas related to the question in complex remote sensing images. This module progressively directs attention from broad areas to finer details through key region routing, enhancing the model's ability to focus on relevant regions. Furthermore, we propose an Adaptive Multi-Expert Fusion (AMEF) module that dynamically integrates different experts, enabling the adaptive fusion of optical and SAR features. In addition, we create the first large-scale benchmark dataset for evaluating optical-SAR RSVQA methods, comprising 6,008 well-aligned optical-SAR image pairs and 1,036,694 well-labeled question-answer pairs across 16 diverse question types, including complex relational reasoning questions. Extensive experiments on the proposed dataset demonstrate that our TGFNet effectively integrates complementary information between optical and SAR images, significantly improving the model's performance in challenging scenarios. The dataset is available at: https://github.com/mmic-lcl/. Index Terms: Remote Sensing Visual Question Answering, Multi-source Data Fusion, Multimodal, Remote Sensing, OPT-SAR.

遥感问答多源融合雷达图像视觉推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。