提出负向学习框架,让弱模态学会不模仿强模态,提升鲁棒性。
Multimodal Negative Learning
- 强模态动态引导弱模态抑制非目标类别,而非强制对齐
- 在噪声和不平衡数据下,弱模态误差显著降低
- 适合处理多模态信息不均衡的场景,如跨模态识别
多模态学习系统常面临模态失衡问题,主导模态会压制弱模态,阻碍其学习。传统方法通过‘正向学习’强制弱模态与主导模态对齐,可能掩盖其独特信息。为此,本文提出新范式——‘负向学习’:不再增强弱模态的目标预测,而是由主导模态动态引导其抑制非目标类。该机制稳定决策空间,保留模态特异性信息。理论推导出多模态负向学习(MNL)框架,通过提升单模态置信度边界(UCoM),可证明性地收紧多模态学习的鲁棒性下界,并在噪声与不平衡场景中显著降低弱模态的实证误差。多个基准测试验证了方法的有效性与通用性。代码将开源至 https://github.com/BaoquanGong/Multimodal-Negative-Learning.git。
原文摘要 · Abstract (English)
Multimodal learning systems often encounter challenges related to modality imbalance, where a dominant modality may overshadow others, thereby hindering the learning of weak modalities. Conventional approaches often force weak modalities to align with dominant ones in "Learning to be (the same)" (Positive Learning), which risks suppressing the unique information inherent in the weak modalities. To address this challenge, we offer a new learning paradigm: "Learning Not to be" (Negative Learning). Instead of enhancing weak modalities' target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes. This stabilizes the decision space and preserves modality-specific information, allowing weak modalities to preserve unique information without being over-aligned. We proceed to reveal multimodal learning from a robustness perspective and theoretically derive the Multimodal Negative Learning (MNL) framework, which introduces a dynamic guidance mechanism tailored for negative learning. Our method provably tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM) and reduces the empirical error of weak modalities, particularly under noisy and imbalanced scenarios. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability of our approach against competing methods. The code will be available at https://github.com/BaoquanGong/Multimodal-Negative-Learning.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。