解决多模态目标检测中融合性能下降问题,提升各模态学习均衡性。
Representation Space Constrained Learning with Modality Decoupling for Multimodal Object Detection
- 通过解耦模块与表征空间约束,缓解多模态融合时梯度抑制问题。
- 在四个数据集上实现领先性能,显著改善弱模态学习不足现象。
- 适合关注多模态模型优化与鲁棒性提升的研究者与工程师。
多模态目标检测因增强的鲁棒性受到学术界与工业界的广泛关注。尽管大量研究聚焦于改进模态融合策略,但多数忽略了融合退化问题,且缺乏对其根本原因的理论分析。本文首次系统性地探究了多模态检测中的融合退化现象,识别出两个关键优化缺陷:(1) 多模态架构下单模态分支的梯度被严重抑制,导致其优化不足;(2) 模态质量差异使弱模态遭受更强梯度抑制,引发模态学习失衡。为此,本文提出代表空间约束学习与模态解耦(RSC-MD)方法,包含两个模块:RSC模块用于放大被抑制的梯度,MD模块用于消除模态间耦合干扰与不平衡问题,从而实现各模态专有主干网络的全面优化。在FLIR、LLVIP、M3FD和MFAD数据集上的大量实验表明,该方法有效缓解融合退化,在多个基准上达到最先进性能。代码与训练流程将发布于https://github.com/yikangshao/RSC-MD。
原文摘要 · Abstract (English)
Multimodal object detection has attracted significant attention in both academia and industry for its enhanced robustness. Although numerous studies have focused on improving modality fusion strategies, most neglect fusion degradation, and none provide a theoretical analysis of its underlying causes. To fill this gap, this paper presents a systematic theoretical investigation of fusion degradation in multimodal detection and identifies two key optimization deficiencies: (1) the gradients of unimodal branch backbones are severely suppressed under multimodal architectures, resulting in under-optimization of the unimodal branches; (2) disparities in modality quality cause weaker modalities to experience stronger gradient suppression, which in turn results in imbalanced modality learning. To address these issues, this paper proposes a Representation Space Constrained Learning with Modality Decoupling (RSC-MD) method, which consists of two modules. The RSC module and the MD module are designed to respectively amplify the suppressed gradients and eliminate inter-modality coupling interference as well as modality imbalance, thereby enabling the comprehensive optimization of each modality-specific backbone. Extensive experiments conducted on the FLIR, LLVIP, M3FD, and MFAD datasets demonstrate that the proposed method effectively alleviates fusion degradation and achieves state-of-the-art performance across multiple benchmarks. The code and training procedures will be released at https://github.com/yikangshao/RSC-MD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。