arXiv:2608.03611cs.AIcs.MM2026-08被引 1

提出显式建模模态可靠性的框架,提升不完整多模态情感分析的准确率。

Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations

论文配图:Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations
图 1 · 摘自论文原文
  • 通过内在质量与跨模态一致性估算每样本的模态可靠性
  • 在交互与融合阶段用可靠性分数动态调节信息流,性能超越现有方法
  • 适合处理真实场景中缺失数据较多的情感分析任务

多模态情感分析(MSA)融合文本、音频和视觉信息推断情感,但实际观测常不完整。现有方法主要分为重建型和联合表示型,但通常隐式处理模态可靠性,导致两个问题:可靠性错配(不同样本间模态证据差异大)与可靠性传播偏差(劣化模态影响跨模态交互)。为此,提出MRCF框架:包含可靠性感知分支(基于模态内质量与跨模态语义一致性估计可靠性)、可靠性引导交互分支(用可靠性调控跨模态信息流)和可靠性校准融合模块(结合可靠性与语义线索预测结果)。在CMU-MOSI、CMU-MOSEI和CH-SIMS数据集上验证,MRCF在标准不完整观测协议下表现优异。分析表明,显式建模可有效缓解可靠性错配与传播偏差。

原文摘要 · Abstract (English)

Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to infer human affect, yet real-world multimodal observations are often incomplete. Existing methods for incomplete-observation MSA mainly follow two paradigms. Reconstruction-based methods recover missing information from observed modalities, while joint-representation methods learn directly from incomplete inputs. Although effective, these methods usually treat modality reliability only implicitly within representation learning or fusion design rather than modeling it explicitly. We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues. The first is reliability mismatch, in which the affective evidence retained by each modality varies across samples and missing rates. The second is reliability propagation bias, in which messages from degraded modalities may adversely affect cross-modal interaction and predictive performance. To address these issues, we propose MRCF, a Modality Reliability-Calibrated Framework for MSA with incomplete observations. MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency, a Reliability-Guided Interaction Branch that uses the estimated scores to modulate cross-modal information flow, and a Reliability-Calibrated Fusion Module that integrates reliability and semantic cues for final prediction. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols. Further analyses provide evidence that explicit reliability modeling helps mitigate reliability mismatch and reliability propagation bias during interaction and fusion.

多模态情感分析可靠性建模缺失数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。