提升弱模态表现，让多模态情感分析更稳定可靠
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
- 先增强后平衡：通过语义解耦和跨模态增强强化弱信号
- 引入能量引导机制，自动调节各模态梯度贡献比例
- 按样本可信度动态调整融合权重，提升缺失模态下的鲁棒性
多模态情感分析（MSA）融合文本、音频和视觉信号以推断人类情绪。尽管近期方法利用跨模态互补性，但常难以充分发挥较弱模态的作用。实际中，主导模态往往压制非语言信号，导致模态竞争，限制整体贡献。这种不平衡会降低融合性能，在噪声或缺失模态下尤为明显。为此，我们提出新型框架EBMC（Enhance-then-Balance Modality Collaboration）。EBMC通过语义解耦与跨模态增强提升表征质量，强化弱模态；采用能量引导的模态协调机制，基于可微均衡目标实现隐式梯度重平衡；同时引入实例感知模态信任蒸馏，估计样本级可靠性并自适应调节融合权重，确保鲁棒性。大量实验表明，EBMC达到领先或相当水平，并在缺失模态设置下保持强性能。
原文摘要 · Abstract (English)
Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。