提出分层表征学习框架,提升模态缺失时的情感分析鲁棒性
Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning
- 通过跨模态翻译与语义重建,分解出情感相关与模态特有表征
- 分层互信息最大化机制提升多尺度表征对齐与高层语义重构
- 对抗学习进一步对齐潜在分布,适合真实场景中不完整数据
多模态情感分析(MSA)旨在通过多种模态理解人类情感,融合互补信息可显著优于单模态。然而在真实应用中,模态缺失常因不可控因素发生,严重影响建模效果。为此,本文提出分层表征学习框架(HRLF),应对不确定的模态缺失问题。首先设计细粒度表征分解模块,通过跨模态翻译与情感语义重建,将各模态分解为情感相关与模态特有表征。其次引入分层互信息最大化机制,逐步提升多尺度表征间的互信息,以对齐并重构高层语义。最后提出分层对抗学习机制,进一步对齐和适配情感相关表征的潜在分布,生成鲁棒的联合多模态表示。在三个数据集上的大量实验表明,该方法在模态缺失情况下显著提升了情感分析性能。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. The complementary information provided by multimodal fusion promotes better sentiment analysis compared to utilizing only a single modality. Nevertheless, in real-world applications, many unavoidable factors may lead to situations of uncertain modality missing, thus hindering the effectiveness of multimodal modeling and degrading the model's performance. To this end, we propose a Hierarchical Representation Learning Framework (HRLF) for the MSA task under uncertain missing modalities. Specifically, we propose a fine-grained representation factorization module that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction. Moreover, a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations. Ultimately, we propose a hierarchical adversarial learning mechanism that further aligns and adapts the latent distribution of sentiment-relevant representations to produce robust joint multimodal representations. Comprehensive experiments on three datasets demonstrate that HRLF significantly improves MSA performance under uncertain modality missing cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。