解决多模态情感分析中文本主导导致的训练不稳问题
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
- 引入冲突感知惩罚机制，实时检测并抑制梯度冲突
- 设计统计损失使预测分布匹配真实输入分布，提升模型稳定性
- 适合关注多模态融合稳定性的研究者和工程师
多模态情感分析（MSA）融合文本、语音和视觉信息以推断情感。由于预训练文本编码器远比语音和视觉编码器表达能力强，文本模态在优化中容易占据主导，抑制较弱模态，引发梯度范数冲突，导致训练不稳定。为此，我们提出冲突感知惩罚（CP），在每一步训练中检测并惩罚梯度范数冲突；同时设计统计损失（SL），使预测分布的统计特性与输入数据的实证统计特性对齐。关键在于，CP 防止主导模态的梯度干扰 SL 目标，实现自适应模态编码、门控跨模态融合与单模态辅助头在统一框架下的协同训练。在 CMU-MOSI 数据集上的实验达到当前最优性能，消融实验验证了各组件的有效性。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) fuses text, acoustic, and visual streams to infer sentiment. Because pre-trained text encoders are far more expressive than their acoustic and visual counterparts, the text modality tends to dominate optimization, suppressing weaker modalities and inducing gradient norm conflicts that destabilize training. To address this, we propose a Conflict-aware Penalty (CP) that detects and penalizes gradient norm conflicts at each training step, and a Statistical Loss (SL) that aligns predicted distribution statistics with empirical input statistics. Crucially, CP prevents dominant modality gradients from interfering with the SL objective, enabling synergistic training within a unified framework incorporating adaptive modality encoding, gated cross-modal fusion, and unimodal auxiliary heads. Experiments on CMU-MOSI demonstrate state-of-the-art performance, with ablation studies confirming the effectiveness of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。