arXiv:2604.05558cs.CV2026-04被引 1

提出新框架,不补全缺失模态也能准确分析多模态情感。

Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities

  • 先评估缺失模态重要性,避免无效数据填补
  • 通过解耦和动态加权,提升模型对缺失的鲁棒性
  • 适合处理真实场景中信息不全的情感分析任务

缺失模态问题是多模态情感分析中的核心挑战,严重影响模型在现实场景下的准确性和泛化能力。现有方法主要依赖提示学习与预训练模型提升鲁棒性,但仍存在两大缺陷:一是生成缺失模态的必要性缺乏严格评估;二是多模态提示间的结构依赖与全局一致性未被充分探索。为此,本文提出基于提示的缺失模态适应框架(ProMMA)。在输入阶段引入缺失模态评估器,利用预训练模型与伪标签动态判断缺失模态的重要性,避免低质量数据补全。进一步设计模态不变提示解耦模块,将共享提示分解为模态专属私有提示,捕捉内在局部相关性以提升表征质量。同时,动态提示加权模块基于跨注意力输出计算互信息权重,自适应抑制缺失模态干扰。为增强全局一致性,引入多层次提示动态连接模块,通过残差连接融合共享提示与自注意力输出,利用全局提示先验强化关键引导特征。在CMU MOSI、CMU MOSEI和CH SIMS三个公开基准上的大量实验表明,该框架在多种缺失模态设置下均达到最优性能且结果稳定。代码已开源:https://github.com/rongfei-chen/ProMMA。

原文摘要 · Abstract (English)

The missing modality problem poses a fundamental challenge in multimodal sentiment analysis, significantly degrading model accuracy and generalization in real world scenarios. Existing approaches primarily improve robustness through prompt learning and pre trained models. However, two limitations remain. First, the necessity of generating missing modalities lacks rigorous evaluation. Second, the structural dependencies among multimodal prompts and their global coherence are insufficiently explored. To address these issues, a Prompt based Missing Modality Adaptation framework is proposed. A Missing Modality Evaluator is introduced at the input stage to dynamically assess the importance of missing modalities using pretrained models and pseudo labels, thereby avoiding low quality data imputation. Building on this, a Modality invariant Prompt Disentanglement module decomposes shared prompts into modality specific private prompts to capture intrinsic local correlations and improve representation quality. In addition, a Dynamic Prompt Weighting module computes mutual information based weights from cross attention outputs to adaptively suppress interference from missing modalities. To enhance global consistency, a Multi level Prompt Dynamic Connection module integrates shared prompts with self attention outputs through residual connections, leveraging global prompt priors to strengthen key guidance features. Extensive experiments on three public benchmarks, including CMU MOSI, CMU MOSEI, and CH SIMS, demonstrate that the proposed framework achieves state of the art performance and stable results under diverse missing modality settings. The implementation is available at https://github.com/rongfei-chen/ProMMA

多模态情感分析缺失数据提示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。