用Whisper提取语音情绪特征，结合注意力池化提升识别效果
Speech Emotion Recognition Leveraging OpenAI's Whisper Representations and Attentive Pooling Methods
- 设计两种注意力池化方法，高效压缩Whisper特征并保留情绪信息
- 在波斯语数据集上实现当前最佳结果，准确率提升2.47%
- 中间层编码器表现更优，适合轻量级情绪识别应用
语音情绪识别（SER）研究受限于缺乏标准且足够大的数据集。近期工作利用预训练模型提取下游任务如SER的特征。本文探索Whisper这一预训练语音识别系统在语音情绪识别中的能力，提出两种基于注意力的池化方法：多头注意力平均池化与QKV池化，旨在高效降低Whisper表示维度的同时保留情绪特征。我们在英文和波斯语数据上分别使用IEMOCAP和ShEMO数据集，采用Whisper Tiny和Small模型进行实验。多头QKV架构在ShEMO数据集上达到当前最优性能，未加权准确率提升2.47%。进一步对比不同Whisper编码层表现，发现对于波斯语数据集，中间层往往优于深层或浅层，提供了一种比HuBERT X-Large等大型模型更轻量高效的替代方案。研究结果表明Whisper可作为有效的特征提取器用于SER，且注意力池化在降维中具有显著优势。
原文摘要 · Abstract (English)
Speech Emotion Recognition (SER) research has faced limitations due to the lack of standard and sufficiently large datasets. Recent studies have leveraged pre-trained models to extract features for downstream tasks such as SER. This work explores the capabilities of Whisper, a pre-trained ASR system, in speech emotion recognition by proposing two attention-based pooling methods, Multi-head Attentive Average Pooling and QKV Pooling, designed to efficiently reduce the dimensionality of Whisper representations while preserving emotional features. We experiment on English and Persian, using the IEMOCAP and ShEMO datasets respectively, with Whisper Tiny and Small. Our multi-head QKV architecture achieves state-of-the-art results on the ShEMO dataset, with a 2.47% improvement in unweighted accuracy. We further compare the performance of different Whisper encoder layers and find that intermediate layers often perform better for SER on the Persian dataset, providing a lightweight and efficient alternative to much larger models such as HuBERT X-Large. Our findings highlight the potential of Whisper as a representation extractor for SER and demonstrate the effectiveness of attention-based pooling for dimension reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。