arXiv:2606.26812cs.CV2026-06中稿 · ECCV

通过掩码引导融合,提升恶劣天气下多模态图像的清晰度与互补性。

Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction

论文配图:Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction
图 1 · 摘自论文原文
  • 用伪真值简化训练,加速特征学习。
  • 设计掩码机制量化各模态贡献,提升融合准确性。
  • 适合自动驾驶、遥感等恶劣环境下的多模态图像处理。

多模态图像融合(MMIF)通过利用不同模态间的互补信息增强场景表征。然而,恶劣天气导致图像严重退化,破坏特征表达,需同时进行特征恢复与跨模态互补。现有方法在该条件下难以有效学习表征,限制实际性能。为此,本文提出一种掩码引导的MMIF方法,集成特征恢复与交互。首先引入“伪真值”简化训练,促进更快更有效的特征学习。其次,基于融合结果与源图像的映射关系设计掩码生成机制,量化融合过程中各模态的相对贡献。通过引入掩码引导的跨模态交叉注意力机制,网络可选择性关注有用特征,降低对“伪真值”静态分布的过拟合风险。此外,提出掩码引导学习与任务耦合退化感知学习策略,平衡特征恢复与交互。在合成与真实数据集上的大量实验表明,本方法在视觉质量、定量指标及下游任务中均优于现有技术。代码已开源:https://github.com/ixilai/AMG-Fuse。

原文摘要 · Abstract (English)

Multi-modality image fusion (MMIF) enhances scene representation by exploiting complementary cues from different modalities. Adverse weather, however, causes significant image degradation, disrupting feature representation and requiring simultaneous feature restoration and cross-modal complementarity. Existing methods often struggle with effective representation learning under such conditions, limiting their practical performance. To address these challenges, we propose a mask-guided MMIF method that integrates feature restoration and interaction. We first introduce "Pseudo Ground Truth" to simplify training, promoting faster and more effective feature learning. Then, we design a mask generation mechanism based on the mapping relationship between the fused result and the source images, quantifying the relative contribution of each modality during the fusion process. By incorporating the proposed mask-guided cross-modal cross-attention mechanism, the network is encouraged to selectively attend to informative features during modality interaction, mitigating the risk of overfitting to the static distribution of the "Pseudo Ground Truth". Additionally, we propose a mask-guided learning strategy and a task-coupled degradation-aware learning strategy to balance feature restoration and interaction. Extensive experiments on synthetic and real-world datasets demonstrate that our method surpasses state-of-the-art approaches in visual quality, quantitative metrics, and downstream tasks. The source code is available at https://github.com/ixilai/AMG-Fuse.

图像融合多模态恶劣天气

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。