解决对话情绪识别中多模态缺失问题,提升模型对不完整数据的鲁棒性。
SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognition
- 构建语义交互图,利用滑动窗口捕捉说话人与上下文关系
- 在频域进行多频率聚合,有效恢复缺失模态特征
- 适合处理真实场景中存在模态缺失的情绪识别任务
对话中的多模态情绪识别(MERC)旨在通过文本、语音和视觉特征分类话语情绪。现有方法通常假设每个话语具备完整模态,忽视了真实场景中常见的模态缺失问题。近年来,图神经网络(GNN)在不完整多模态对话情绪识别(IMERC)中取得显著进展。然而,传统GNN仅关注节点间的二元关系,难以捕捉更复杂的高阶信息;且重复消息传递易导致过平滑,削弱对高频细节的保留能力。为此,本文提出频域重建图神经网络(SDR-GNN),用于不完整多模态学习。SDR-GNN基于滑动窗口构建话语语义交互图,结合说话人与上下文关系以建模情绪依赖。为捕捉高阶与高频信息,采用加权关系聚合,确保跨话语特征提取一致性。同时,在频域执行多频率聚合,通过提取高低频信息高效恢复缺失模态。最后,使用多头注意力融合并优化特征以提升情绪识别性能。在多个真实世界数据集上的大量实验表明,该方法在不完整多模态学习中表现优异,优于当前最先进方法。
原文摘要 · Abstract (English)
Multimodal Emotion Recognition in Conversations (MERC) aims to classify utterance emotions using textual, auditory, and visual modal features. Most existing MERC methods assume each utterance has complete modalities, overlooking the common issue of incomplete modalities in real-world scenarios. Recently, graph neural networks (GNNs) have achieved notable results in Incomplete Multimodal Emotion Recognition in Conversations (IMERC). However, traditional GNNs focus on binary relationships between nodes, limiting their ability to capture more complex, higher-order information. Moreover, repeated message passing can cause over-smoothing, reducing their capacity to preserve essential high-frequency details. To address these issues, we propose a Spectral Domain Reconstruction Graph Neural Network (SDR-GNN) for incomplete multimodal learning in conversational emotion recognition. SDR-GNN constructs an utterance semantic interaction graph using a sliding window based on both speaker and context relationships to model emotional dependencies. To capture higher-order and high-frequency information, SDR-GNN utilizes weighted relationship aggregation, ensuring consistent semantic feature extraction across utterances. Additionally, it performs multi-frequency aggregation in the spectral domain, enabling efficient recovery of incomplete modalities by extracting both high- and low-frequency information. Finally, multi-head attention is applied to fuse and optimize features for emotion recognition. Extensive experiments on various real-world datasets demonstrate that our approach is effective in incomplete multimodal learning and outperforms current state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。