arXiv:2608.09986cs.AIcs.LG2026-08TPAMI

提出MIDAS框架,解决多模态情感分析中模态缺失问题。

MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis

论文配图:MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis
图 1 · 摘自论文原文
  • 用变分模型分解模态为共享与独有因子,实现信息解耦。
  • 在三种数据集上表现优于基线,不同缺失率下均稳定提升。
  • 通过后验方差自适应加权,提升不完整数据融合鲁棒性。

现有大多数多模态情感分析方法假设输入模态完整,但实际应用中常出现模态缺失或损坏。尽管已有方法尝试应对,主要依赖数据补全和启发式约束,难以有效提取并利用不完整数据中的任务相关特征。为此,我们提出统一框架MIDAS(互信息解耦与不确定性感知融合),在模态缺失条件下重构多模态表示。MIDAS采用变分建模策略,将各模态表示为多变量高斯潜变量,并进一步分解为共享与独有因子。为获得可靠表示,设计最小-最大目标:最小化共享与独有空间间的互信息以实现稳定解耦,同时最大化跨模态共享空间间的互信息以增强语义对齐。此外,引入不确定性感知融合机制,利用后验方差作为可靠性指标,自适应加权潜变量进行融合,确保在模态缺失时仍具鲁棒性。在三个常用数据集上的大量实验表明,MIDAS在多种缺失设置下显著优于对比基线,验证了其在不完整数据场景下的有效性与鲁棒性。

原文摘要 · Abstract (English)

Most existing multimodal sentiment analysis approaches assume access to complete multimodal inputs. However, real-world applications frequently encounter incomplete or corrupted modalities, posing a critical challenge. Although several methods have been proposed to tackle this issue, they mainly rely on data imputation and heuristic coordination constraints, which fail to effectively extract and leverage task-relevant information from the incomplete multimodal data. To address this challenge, we propose a unified framework termed Mutual Information Disentanglement with uncertainty-Aware fuSion (MIDAS), which effectively restructures multimodal representations under incomplete conditions. MIDAS adopts a variational modeling strategy to represent each modality with multivariate Gaussian latent variables and further decomposes them into shared and exclusive factors. To obtain reliable representations, we design a minimax objective that minimizes the mutual information between shared and exclusive spaces for stable disentanglement, while maximizing the mutual information among shared spaces across modalities to enhance semantic alignment. In addition, an uncertainty-aware fusion mechanism is introduced, where posterior variance is leveraged as a reliability indicator to adaptively weight latent features during fusion, ensuring robust integration even when modalities are incomplete. Extensive experiments on three widely used datasets show that MIDAS achieves strong and consistent performance gains over competitive baselines across a wide range of incomplete settings, demonstrating its effectiveness and robustness for incomplete data scenarios.

多模态情感分析解耦缺失数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。