利用情绪标签信息提升多模态情感识别准确率
Leveraging Label Potential for Enhanced Multimodal Emotion Recognition
- 通过标签信号增强模块融合标签嵌入与音视频特征
- 在IEMOCAP和MELD数据集上实现更高分类准确率
- 适合关注情感分析与跨模态融合的研究者
多模态情感识别（MER）旨在融合多种模态以准确预测情感状态。然而，当前多数研究仅聚焦于音频与文本特征的融合，忽略了情绪标签中蕴含的丰富信息。这一忽略可能限制现有方法的表现，因为情绪标签包含有助于提升MER的深层洞察。本文提出一种名为标签信号引导的多模态情感识别（LSGMER）的新模型，旨在充分挖掘情绪标签信息以提升分类准确率与稳定性。具体而言，LSGMER引入标签信号增强模块，通过标签嵌入与音视频特征交互优化模态表示，精准捕捉情绪细微差别。此外，提出联合目标优化（JOO）方法，引入归因-预测一致性约束（APC），强化融合特征与情感类别间的对齐。在IEMOCAP和MELD数据集上的大量实验验证了该模型的有效性。
原文摘要 · Abstract (English)
Multimodal emotion recognition (MER) seeks to integrate various modalities to predict emotional states accurately. However, most current research focuses solely on the fusion of audio and text features, overlooking the valuable information in emotion labels. This oversight could potentially hinder the performance of existing methods, as emotion labels harbor rich, insightful information that could significantly aid MER. We introduce a novel model called Label Signal-Guided Multimodal Emotion Recognition (LSGMER) to overcome this limitation. This model aims to fully harness the power of emotion label information to boost the classification accuracy and stability of MER. Specifically, LSGMER employs a Label Signal Enhancement module that optimizes the representation of modality features by interacting with audio and text features through label embeddings, enabling it to capture the nuances of emotions precisely. Furthermore, we propose a Joint Objective Optimization(JOO) approach to enhance classification accuracy by introducing the Attribution-Prediction Consistency Constraint (APC), which strengthens the alignment between fused features and emotion categories. Extensive experiments conducted on the IEMOCAP and MELD datasets have demonstrated the effectiveness of our proposed LSGMER model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。