缺文本时仍能精准做多模态情感分析,靠大模型模拟+跨模态对齐。
Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention
- 用大模型从音频生成模拟文本,辅以跨模态信息补全。
- 在CMU-MOSEI上缺文本时MAE低于0.24,显著优于基线。
- 适合低文本标注成本场景,如语音情感分析应用。
多模态情感分析中,文本数据因标注成本高和自动语音识别(ASR)质量不一而更难获取。为应对这一挑战,本文提出双流自蒸馏框架,包含统一模态交叉注意力(UMCA)与模态想象自编码器(MIA),可有效处理完整模态与缺失文本两种情况。当文本缺失时,基于LLM的模型从音频生成模拟文本表示,MIA模块融合其他两模态信息,使模拟文本接近真实文本表示。为进一步对齐模拟与真实表示,并捕捉情感强度回归任务中样本顺序的连续性,引入排名对比损失(RNC)。在CMU-MOSEI数据集测试中,该模型在缺失文本情况下达到低于0.24的平均绝对误差(MAE),显著优于现有方法。代码已开源。
原文摘要 · Abstract (English)
In multimodal sentiment analysis, collecting text data is often more challenging than video or audio due to higher annotation costs and inconsistent automatic speech recognition (ASR) quality. To address this challenge, our study has developed a robust model that effectively integrates multimodal sentiment information, even in the absence of text modality. Specifically, we have developed a Double-Flow Self-Distillation Framework, including Unified Modality Cross-Attention (UMCA) and Modality Imagination Autoencoder (MIA), which excels at processing both scenarios with complete modalities and those with missing text modality. In detail, when the text modality is missing, our framework uses the LLM-based model to simulate the text representation from the audio modality, while the MIA module supplements information from the other two modalities to make the simulated text representation similar to the real text representation. To further align the simulated and real representations, and to enable the model to capture the continuous nature of sample orders in sentiment valence regression tasks, we have also introduced the Rank-N Contrast (RNC) loss function. When testing on the CMU-MOSEI, our model achieved outstanding performance on MAE and significantly outperformed other models when text modality is missing. The code is available at: https://github.com/WarmCongee/SDUMC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。