用频率分解提升红外与可见光图像融合，改善复杂光照下语义分割效果。
Frequency-Guided Fusion For RGB-Thermal Semantic Segmentation
- 通过频域分解和双分支注意力，有选择地融合红外图像的粗纹理与细边界信息。
- 在MFNet和PST900数据集上分别达到61.73%和86.24% mIoU，参数仅35.43M。
- 适合做轻量级多模态感知系统，尤其适用于自动驾驶场景的视觉理解。
在城市驾驶等复杂环境下，恶劣光照条件使仅靠可见光图像进行语义分割仍具挑战性。红外与可见光图像的互补特性可提升场景理解能力，但如何在不同抽象层次有效融合异构模态仍是未解难题。本文提出一种基于双ConvNeXt V2主干网络的多模态融合架构，采用分阶段、模态自适应的融合策略。早期特征阶段引入基于频率的融合模块，通过高斯滤波将红外特征分解为低频与高频成分，使用双分支空间注意力机制选择性增强热图模式与细粒度边界，并通过置信度门控残差机制与可见光特征融合。晚期特征阶段设计跨模态注意力与多尺度深度卷积的语义融合模块，以捕捉模态间语义对应关系。融合特征由带有深度监督的PANet式双向解码器重建。在MFNet和PST900数据集上的实验表明，最轻量版本分别取得61.73%和86.24%的mIoU，仅需35.43M参数，优于现有方法且计算开销更低。代码已开源。
原文摘要 · Abstract (English)
Semantic segmentation in complex environments such as urban driving scenes remains challenging under adverse lighting conditions, where RGB images alone provide insufficient information. RGB-Thermal fusion leverages the complementary strengths of visible and infrared imagery to improve scene understanding; however, effectively integrating these heterogeneous modalities at varying levels of feature abstraction remains an open problem. In this paper, we propose a multi-modal fusion architecture built upon dual ConvNeXt V2 backbones that employs stage-wise, modality-adaptive fusion strategies. For early-stage features, we introduce a Frequency-Based Fusion Module that decomposes infrared features into low- and high-frequency components via Gaussian filtering, applies dual-branch spatial attention to selectively emphasize thermal patterns and fine-grained boundaries, and integrates them with RGB features through a confidence-gated residual mechanism. For late-stage features, we design a semantic fusion module with cross-modal attention and multi-scale depthwise convolutions to capture semantic correspondences across modalities. The fused features are decoded via a PANet-style bidirectional decoder with deep supervision. Experiments on MFNet and PST900 demonstrate that our lightest variant achieves 61.73\% and 86.24\% mIoU, respectively, with only 35.43M parameters, outperforming recent methods while using substantially fewer parameters and lower computational cost. Code is available at https://github.com/ismailemrecntz/VISIBLE-INFRARED-SENSOR-FUSION
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。