融合红外与可见光图像，提升大模型在恶劣环境下的视觉推理能力
DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning
- 设计轻量级跨模态注意力模块，实现红外与可见光图像的像素级融合
- 在204K标注数据上验证，模型在雾霾、低照度等退化条件下性能显著提升
- 开源数据集与代码，适合做多模态鲁棒性研究的学者使用
多模态大语言模型（MLLMs）在可见光图像的视觉感知与推理任务中表现优异，但在雾霾、模糊或低光照等常见退化条件下仍显脆弱。红外（IR）成像作为可见光的可靠补充，在此类环境下具有天然优势，但其在MLLM中的应用尚未充分探索。为此，我们提出DUALVISION，一种轻量级融合模块，通过局部化跨注意力机制在图像块级别高效整合红外与可见光信息。为支持训练与评估，我们还构建了DV-204K数据集，包含约25,000对对齐的红外-可见光图像及204,000条模态特异性问答标注；以及DV-500基准，含500对红外-可见光图像与500组问答，用于评估跨模态推理能力。基于这些数据，我们对开源与闭源的MLLM进行了评测，结果表明DUALVISION在多种视觉退化场景下均表现出色。相关代码与数据集已公开于https://abrarmajeedi.github.io/dualvision。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved impressive performance on visual perception and reasoning tasks with RGB imagery, yet they remain fragile under common degradations, such as fog, blur, or low-light conditions. Infrared (IR) imaging, a well-established complement to RGB, offers inherent robustness in these conditions, but its integration into MLLMs remains underexplored. To bridge this gap, we propose DUALVISION, a lightweight fusion module that efficiently incorporates IR-RGB information into MLLMs via patch-level localized cross-attention. To support training and evaluation and to facilitate future research, we also introduce DV-204K, a dataset of ~25K publicly available aligned IR-RGB image pairs with 204K modality-specific QA annotations, and DV-500, a benchmark of 500 IR-RGB image pairs with 500 QA pairs designed for evaluating cross-modal reasoning. Leveraging these datasets, we benchmark both open- and closed-source MLLMs and demonstrate that DUALVISION delivers strong empirical performance under a wide range of visual degradations. Our code and dataset are available at https://abrarmajeedi.github.io/dualvision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。