TUNI统一预训练与微调,提升红外可见光语义分割精度与效率。
TUNI: Unifying Pre-training and Fine-tuning with Modality-Aware Mutual Learning and Rectification for RGB-T Semantic Segmentation
- 通过模态感知互学习与校正,统一处理多模态特征提取与融合
- 在5个数据集上超越15个主流模型,精度更高且模型更紧凑
- 适用于自动驾驶等需鲁棒环境感知的实时场景
红外-可见光(RGB-T)语义分割能提升自动驾驶平台在复杂环境下的感知能力。现有框架存在多模态特征提取与融合不佳、模态依赖失衡及热成像信息利用不足等问题。为此,我们提出TUNI,一种统一的预训练与微调框架,实现高效实时的RGB-T语义分割。其预训练阶段采用包含跨模态局部模块的编码器,选择性强调模态间一致与差异的显著局部特征,统一完成跨模态特征提取与融合。为缓解预训练中的模态偏差,引入模态反转对比互学习,促进两个以可见光为主和热成像为主的编码器间知识交换。微调阶段,通过模态校正学习聚焦于两模态解码器预测差异但正确的区域,充分挖掘残余热成像信息。我们进一步设计三种TUNI变体,覆盖轻量、平衡与高性能需求。在五个RGB-T语义分割数据集上的大量实验表明,TUNI相比15个先进模型,在精度、泛化性和紧凑性方面均表现更优。代码已开源:https://github.com/xiaodonguo/TUNI-v2。
原文摘要 · Abstract (English)
RGB-thermal (RGB-T) semantic segmentation improves the environmental perception of autonomous platforms in challenging conditions. Prevailing RGB-T segmentation frameworks suffer from suboptimal multi-modal feature extraction and fusion, unbalanced modality dependency, and inadequate utilization of thermal information. To address these challenges, we propose TUNI, a unified pre-training and fine-tuning framework for efficient and real-time RGB-T semantic segmentation. It pre-trains an RGB-T encoder that incorporates an RGB-T local module that selectively emphasizes salient consistent and distinct local features across modalities, thereby integrating cross-modal feature extraction and fusion in a unified manner. To alleviate the modality bias issue during RGB-T pre-training, modality-inverted contrastive mutual learning is introduced to enable knowledge exchange between two RGB-dominated and thermal-dominated encoders. In the fine-tuning phase, modality rectification learning fully exploits residual thermal information by focusing on correct yet divergent prediction regions between two modality-specific decoders. We further develop three TUNI variants, covering lightweight, balanced, and high-performance requirements. Extensive experiments on five RGB-T semantic segmentation datasets demonstrate that TUNI achieves superior accuracy, generalization, and compactness compared with 15 state-of-the-art models. The code is available at https://github.com/xiaodonguo/TUNI-v2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。