通过最大化多模态特征与标签的总相关性,缓解模态竞争问题。
Multimodal Classification via Total Correlation Maximization
- 从信息论角度建模模态间关系,用总相关性约束特征对齐
- 提出TCMax损失函数,在多个数据集上超越现有联合与单模学习方法
- 无需调参,适合多模态融合场景中弱模态被忽略的问题
多模态学习通过整合来自不同传感器的数据来充分利用各模态信息。然而,近期研究表明,联合学习常过度依赖某些模态而忽视其他模态,导致性能劣于单模学习。尽管已有工作尝试平衡模态贡献或结合联合与单模学习以缓解弱模态退化问题并取得良好效果,但鲜有研究从信息论视角分析联合与单模学习的关系。本文从理论上分析模态竞争现象,提出一种通过最大化多模态特征与标签之间总相关性的多模态分类方法。通过最大化该目标,所提方法在缓解模态竞争的同时,通过特征对齐捕捉模态间交互。基于互信息神经估计(MINE),我们引入总相关性神经估计(TCNE)以推导总相关性的下界,并进一步提出无超参数的TCMax损失函数,通过变分界优化实现总相关性最大化。大量实验表明,TCMax优于当前主流的联合与单模学习方法。代码已公开于https://github.com/hubaak/TCMax。
原文摘要 · Abstract (English)
Multimodal learning integrates data from diverse sensors to effectively harness information from different modalities. However, recent studies reveal that joint learning often overfits certain modalities while neglecting others, leading to performance inferior to that of unimodal learning. Although previous efforts have sought to balance modal contributions or combine joint and unimodal learning, thereby mitigating the degradation of weaker modalities with promising outcomes, few have examined the relationship between joint and unimodal learning from an information-theoretic perspective. In this paper, we theoretically analyze modality competition and propose a method for multimodal classification by maximizing the total correlation between multimodal features and labels. By maximizing this objective, our approach alleviates modality competition while capturing inter-modal interactions via feature alignment. Building on Mutual Information Neural Estimation (MINE), we introduce Total Correlation Neural Estimation (TCNE) to derive a lower bound for total correlation. Subsequently, we present TCMax, a hyperparameter-free loss function that maximizes total correlation through variational bound optimization. Extensive experiments demonstrate that TCMax outperforms state-of-the-art joint and unimodal learning approaches. Our code is available at https://github.com/hubaak/TCMax.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。