arXiv:2508.16881cs.CV2025-08被引 6

用文字描述提升恶劣天气图像融合效果,让画面更清晰真实。

AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perception

  • 用全局文字感知识别天气类型,局部文字聚焦具体问题
  • 结合BLIP和ChatGPT生成描述,显著提升复杂天气下融合质量
  • 适合需要高语义准确性的自动驾驶与遥感图像处理场景

恶劣天气下的多模态图像融合旨在解决气象退化导致的视觉信息丢失,提供更清晰的场景表示。尽管已有研究尝试引入文本信息以增强语义感知,但普遍存在文本内容分类不充分、分析不深入的问题。为此,我们提出AWM-Fuse,一种基于全局与局部文本感知的新型融合方法,采用统一共享权重架构,有效应对多种退化情况。全局特征感知模块利用BLIP生成的标题提取整体场景特征,并识别主要退化类型,提升在不同恶劣天气下的泛化能力;局部模块则借助ChatGPT生成的详细场景描述,通过具体文本线索聚焦特定退化效应,捕捉更细微细节。此外,文本描述被用于约束融合图像生成,引导网络学习更符合真实语义标签的视觉特征,促进有意义视觉表示的学习。大量实验表明,AWM-Fuse在复杂天气条件及下游任务中均优于当前最先进方法。代码已公开于https://github.com/Feecuin/AWM-Fuse。

原文摘要 · Abstract (English)

Multi-modality image fusion (MMIF) in adverse weather aims to address the loss of visual information caused by weather-related degradations, providing clearer scene representations. Although less studies have attempted to incorporate textual information to improve semantic perception, they often lack effective categorization and thorough analysis of textual content. In response, we propose AWM-Fuse, a novel fusion method for adverse weather conditions, designed to handle multiple degradations through global and local text perception within a unified, shared weight architecture. In particular, a global feature perception module leverages BLIP-produced captions to extract overall scene features and identify primary degradation types, thus promoting generalization across various adverse weather conditions. Complementing this, the local module employs detailed scene descriptions produced by ChatGPT to concentrate on specific degradation effects through concrete textual cues, thereby capturing finer details. Furthermore, textual descriptions are used to constrain the generation of fusion images, effectively steering the network learning process toward better alignment with real semantic labels, thereby promoting the learning of more meaningful visual features. Extensive experiments demonstrate that AWM-Fuse outperforms current state-of-the-art methods in complex weather conditions and downstream tasks. Our code is available at https://github.com/Feecuin/AWM-Fuse.

图像融合文本感知恶劣天气

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。