让视觉语言模型学会看红外热图，提升黑暗环境下的理解能力
Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
- 用可训练的热成像编码器和文本引导的双注意力融合模块，实现多光谱信息融合
- 在低光场景下，热红外与可见光联合推理性能显著提升，热图贡献率达32%
- 适合需要跨谱段感知的自动驾驶、安防监控等场景
视觉语言模型（VLM）在低光照条件下常因依赖可见光图像而失效，而热红外在可见线索退化时仍能保留互补的场景结构。我们提出Thermo-VL，一种波长感知的VLM，基于冻结的Molmo-7B主干网络，引入可训练的热成像编码器与文本引导的双注意力融合模块。给定对齐的RGB tokens、热力图tokens及提示嵌入，融合模块将热特征同时受语言与RGB上下文条件约束，并通过门控残差注入冻结的RGB流，使热信号融入而不破坏预训练的RGB-语言接口。模型采用标准语言建模目标，辅以对齐与正则化损失，增强跨模态对齐并减少对可见光的过度依赖。我们还构建了像素级对齐的RGB-热指令微调数据集，以及人工筛选的多光谱问答基准Thermo-VL-Bench。实验表明，在复杂热图独立与多模态推理任务中均有显著提升，凸显提示条件多谱融合的价值。代码与数据集已公开。
原文摘要 · Abstract (English)
Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: https://thusharakart.github.io/Thermo-VL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。