arXiv:2603.09149cs.CV2026-03被引 1

提出统一融合与解耦的RGB-T分割模型,提升传感器缺失时的鲁棒性。

RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation

  • 三分支结构同步融合与解耦,实现端到端训练。
  • 在模态缺失下仍保持稳定性能,尤其在热成像缺失时优于现有方法。
  • 适合机器人在暗光环境下部署,无需复杂多阶段训练。

RGB-热成像(RGB-T)语义分割对低光或黑暗环境中运行的机器人系统至关重要。传统方法过度强调模态平衡,导致传感器信号部分缺失时鲁棒性差、性能显著下降。近期方法如跨模态知识蒸馏和模态自适应微调虽增强跨模态交互,但通常将模态融合与模态适应分离,需多阶段训练且依赖冻结模型或师生框架。本文提出RTFDNet,一种三分支编码器-解码器结构,统一融合与解耦以实现鲁棒的RGB-T分割。协同特征融合(SFF)通过通道门控交换和轻量级空间注意力注入互补信息。跨模态解耦正则化(CMDR)从融合表示中分离出模态特异性成分,并通过停止梯度的目标监督单模态解码器。区域解耦正则化(RDR)在置信区域强制类别选择性预测一致性,同时阻断梯度回传至融合分支。该反馈机制强化单模态路径而不损害融合流,支持测试时高效独立推理。大量实验验证了其有效性,在不同模态条件下均表现一致。代码已公开于https://github.com/curapima/RTFDNet。

原文摘要 · Abstract (English)

RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light or dark environments. However, traditional approaches often overemphasize modality balance, resulting in limited robustness and severe performance degradation when sensor signals are partially missing. Recent advances such as cross-modal knowledge distillation and modality-adaptive fine-tuning attempt to enhance cross-modal interaction, but they typically decouple modality fusion and modality adaptation, requiring multi-stage training with frozen models or teacher-student frameworks. We present RTFDNet, a three-branch encoder-decoder that unifies fusion and decoupling for robust RGB-T segmentation. Synergistic Feature Fusion (SFF) performs channel-wise gated exchange and lightweight spatial attention to inject complementary cues. Cross-Modal Decouple Regularization (CMDR) isolates modality-specific components from the fused representation and supervises unimodal decoders via stop-gradient targets. Region Decouple Regularization (RDR) enforces class-selective prediction consistency in confident regions while blocking gradients to the fusion branch. This feedback loop strengthens unimodal paths without degrading the fused stream, enabling efficient standalone inference at test time. Extensive experiments demonstrate the effectiveness of RTFDNet, showing consistent performance across varying modality conditions. Our implementation will be released to facilitate further research. Our source code are publicly available at https://github.com/curapima/RTFDNet.

RGB-T分割多模态融合鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。