解决多模态情感分析中模态竞争问题，提升各模态协作效果
PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis
- 通过原型对齐校准优化单模态表示，增强语义一致性
- 引入基于谢尔普利值的梯度调节，动态平衡各模态贡献
- 在IEMOCAP/MOSI/MOSEI上性能领先，有效缓解模态压制现象
多模态情感分析（MSA）旨在融合文本、语音和视觉信号以理解人类情绪。尽管多模态融合旨在利用跨模态互补性，但现实场景中常出现模态竞争：主导模态往往掩盖较弱模态，导致性能下降。本文提出PaSE框架，即原型对齐校准与谢尔普利值均衡机制，强化模态间协作并显式缓解模态竞争。PaSE首先采用原型引导校准学习（PCL），通过熵最优传输机制优化单模态表示并实现语义对齐；为稳定优化，引入双阶段策略：先用原型门控融合模块提取共享表示，再通过谢尔普利值梯度调制（SGM）根据各模态贡献自适应调整梯度。在IEMOCAP、MOSI和MOSEI数据集上的大量实验表明，PaSE显著优于现有方法，并有效缓解模态竞争问题。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) seeks to understand human emotions by integrating textual, acoustic, and visual signals. Although multimodal fusion is designed to leverage cross-modal complementarity, real-world scenarios often exhibit modality competition: dominant modalities tend to overshadow weaker ones, leading to suboptimal performance. In this paper, we propose PaSE, a novel Prototype-aligned Calibration and Shapley-optimized Equilibrium framework, which enhances collaboration while explicitly mitigating modality competition. PaSE first applies Prototype-guided Calibration Learning (PCL) to refine unimodal representations and align them through an Entropic Optimal Transport mechanism that ensures semantic consistency. To further stabilize optimization, we introduce a Dual-Phase Optimization strategy. A prototype-gated fusion module is first used to extract shared representations, followed by Shapley-based Gradient Modulation (SGM), which adaptively adjusts gradients according to the contribution of each modality. Extensive experiments on IEMOCAP, MOSI, and MOSEI confirm that PaSE achieves the superior performance and effectively alleviates modality competition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。