通过知识蒸馏增强弱模态表示，提升对话情感识别性能
Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation
- 用提示学习强化文本模态，知识蒸馏提升弱模态特征
- 在IEMOCAP和MELD上达到当前最优准确率
- 适合关注多模态融合与模型压缩的语音/情感研究者
对话中的情感识别（ERC）旨在检测对话中每条语句的情感。为每条语句生成高效且模态特异的表示仍是重大挑战。以往研究虽尝试整合不同模态编码器提取的特征，但忽略了各模态对任务贡献的差异，并在帧级别对齐模态导致复杂度升高。为此，我们提出多模态锚点门控变压器结合知识蒸馏（MAGTKD）模型解决该问题。具体地，采用提示学习增强文本模态表示，利用知识蒸馏强化表现较弱的模态；同时引入多模态锚点门控变压器，有效融合跨模态的语句级表示。在IEMOCAP和MELD数据集上的大量实验表明，知识蒸馏显著提升了模态表示能力，实现了对话情感识别的最先进性能。代码已公开于：https://github.com/JieLi-dd/MAGTKD。
原文摘要 · Abstract (English)
Emotion Recognition in Conversation (ERC) aims to detect the emotions of individual utterances within a conversation. Generating efficient and modality-specific representations for each utterance remains a significant challenge. Previous studies have proposed various models to integrate features extracted using different modality-specific encoders. However, they neglect the varying contributions of modalities to this task and introduce high complexity by aligning modalities at the frame level. To address these challenges, we propose the Multi-modal Anchor Gated Transformer with Knowledge Distillation (MAGTKD) for the ERC task. Specifically, prompt learning is employed to enhance textual modality representations, while knowledge distillation is utilized to strengthen representations of weaker modalities. Furthermore, we introduce a multi-modal anchor gated transformer to effectively integrate utterance-level representations across modalities. Extensive experiments on the IEMOCAP and MELD datasets demonstrate the effectiveness of knowledge distillation in enhancing modality representations and achieve state-of-the-art performance in emotion recognition. Our code is available at: https://github.com/JieLi-dd/MAGTKD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。