arXiv:2412.16444cs.CLcs.LG2024-12被引 17

用图模型提升对话情绪识别,兼顾多模态与上下文关系。

Effective Context Modeling Framework for Emotion Recognition in Conversations

  • 构建双并行图结构,分别捕捉话语影响与多模态关联。
  • 在IEMOCAP和MELD上达到当前最优性能,优于已有方法。
  • 针对少数类情绪有效重加权,提升小样本情绪识别能力。

对话中的情绪识别(ERC)有助于深入理解说话人在每条语句中表达的情绪。近年来,图神经网络(GNN)在捕捉数据关系方面表现出色,尤其在上下文建模和多模态融合中。然而,现有方法难以充分建模多模态间及对话上下文的复杂交互,限制了其表达能力。为此,我们提出ConxGNN,一种基于GNN的新型框架,用于捕捉对话中的上下文信息。ConxGNN包含两个关键并行模块:一个多尺度异构图,用于捕捉话语对情绪变化的多样化影响;一个超图,用于建模多模态与话语间的多变量关系。这两个模块的输出通过融合层整合,并应用跨模态注意力机制,生成上下文增强的表示。此外,为解决少数类或语义相似情绪类别的识别难题,我们在损失函数中引入重加权策略。在IEMOCAP和MELD基准数据集上的实验结果表明,该方法显著优于现有基线,达到当前最优性能。

原文摘要 · Abstract (English)

Emotion Recognition in Conversations (ERC) facilitates a deeper understanding of the emotions conveyed by speakers in each utterance within a conversation. Recently, Graph Neural Networks (GNNs) have demonstrated their strengths in capturing data relationships, particularly in contextual information modeling and multimodal fusion. However, existing methods often struggle to fully capture the complex interactions between multiple modalities and conversational context, limiting their expressiveness. To overcome these limitations, we propose ConxGNN, a novel GNN-based framework designed to capture contextual information in conversations. ConxGNN features two key parallel modules: a multi-scale heterogeneous graph that captures the diverse effects of utterances on emotional changes, and a hypergraph that models the multivariate relationships among modalities and utterances. The outputs from these modules are integrated into a fusion layer, where a cross-modal attention mechanism is applied to produce a contextually enriched representation. Additionally, ConxGNN tackles the challenge of recognizing minority or semantically similar emotion classes by incorporating a re-weighting scheme into the loss functions. Experimental results on the IEMOCAP and MELD benchmark datasets demonstrate the effectiveness of our method, achieving state-of-the-art performance compared to previous baselines.

情绪识别图神经网络多模态对话系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。