仅用语音就能精准识别对话情绪，靠的是预训练语音模型自带的语义与情感特征。
Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations
- 利用预训练语音模型的对齐特征，纯音频实现情绪分类
- 在IEMOCAP上比基线模型高3.0%（单模态）、1.2%（多模态）
- 适合语音情绪识别、少数据场景下的高效模型设计
情绪识别在智能人机交互中至关重要。多模态方法通过融合多种信息提升准确率，但高质量多模态数据稀缺且模态对齐困难，限制了其性能。本文提出的Qieemo框架，利用预训练自动语音识别（ASR）模型的骨干网络，该网络天然包含帧级对齐的文本与情感特征，仅基于音频模态即可实现精确情绪分类。此外，设计了多模态融合（MMF）模块和跨模态注意力（CMA）模块，融合ASR编码器提取的音素后验图（PPG）与情感特征以进一步提升识别精度。在IEMOCAP数据集上的实验表明，Qieemo在单模态、多模态及自监督模型上分别取得3.0%、1.2%、1.9%的绝对提升。
原文摘要 · Abstract (English)
Emotion recognition plays a pivotal role in intelligent human-machine interaction systems. Multimodal approaches benefit from the fusion of diverse modalities, thereby improving the recognition accuracy. However, the lack of high-quality multimodal data and the challenge of achieving optimal alignment between different modalities significantly limit the potential for improvement in multimodal approaches. In this paper, the proposed Qieemo framework effectively utilizes the pretrained automatic speech recognition (ASR) model backbone which contains naturally frame aligned textual and emotional features, to achieve precise emotion classification solely based on the audio modality. Furthermore, we design the multimodal fusion (MMF) module and cross-modal attention (CMA) module in order to fuse the phonetic posteriorgram (PPG) and emotional features extracted by the ASR encoder for improving recognition accuracy. The experimental results on the IEMOCAP dataset demonstrate that Qieemo outperforms the benchmark unimodal, multimodal, and self-supervised models with absolute improvements of 3.0%, 1.2%, and 1.9% respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。