解决红外与可见光融合中的主导模态偏移问题，提升多模态感知效果。
Modality Dominance-Aware Optimization for Embodied RGB-Infrared Perception
- 提出模态主导指数MDI，量化不同模态的信息优势。
- 引入HCG与AER机制，平衡跨模态优化动态，实现更优融合。
- 在三个基准上达到最新最优，适合复杂环境下的多模态系统研究。
RGB-Infrared（RGB-IR）多模态感知是复杂物理环境中具身多媒体系统的核心。尽管近期交叉模态融合方法已显著提升RGB-IR检测性能，但由模态特性不对称引发的优化动态仍缺乏深入研究。实际中，信息密度与特征质量的差异导致持续的优化偏差，使训练过度依赖主导模态，阻碍有效融合。为此，我们提出模态主导指数（MDI），通过联合建模特征熵与梯度贡献来量化模态主导程度。基于MDI，我们构建了模态主导感知交叉学习框架（MDACL），包含分层交叉模态引导（HCG）以增强特征对齐，以及对抗均衡正则化（AER）以平衡融合过程中的优化动态。在三个RGB-IR基准上的大量实验表明，MDACL能有效缓解优化偏差，并取得当前最优性能。
原文摘要 · Abstract (English)
RGB-Infrared (RGB-IR) multimodal perception is fundamental to embodied multimedia systems operating in complex physical environments. Although recent cross-modal fusion methods have advanced RGB-IR detection, the optimization dynamics caused by asymmetric modality characteristics remain underexplored. In practice, disparities in information density and feature quality introduce persistent optimization bias, leading training to overemphasize a dominant modality and hindering effective fusion. To quantify this phenomenon, we propose the Modality Dominance Index (MDI), which measures modality dominance by jointly modeling feature entropy and gradient contribution. Based on MDI, we develop a Modality Dominance-Aware Cross-modal Learning (MDACL) framework that regulates cross-modal optimization. MDACL incorporates Hierarchical Cross-modal Guidance (HCG) to enhance feature alignment and Adversarial Equilibrium Regularization (AER) to balance optimization dynamics during fusion. Extensive experiments on three RGB-IR benchmarks demonstrate that MDACL effectively mitigates optimization bias and achieves SOTA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。