arXiv:2508.14130eess.AScs.LG2025-08被引 5

用少量参数让大模型高效识别语音情绪,效果优于多数现有方法。

EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition

  • 通过可学习接口将音频特征映射到大模型空间,融合文本与语音输入。
  • 在标准数据集上性能接近顶尖模型,参数量不到对手一半。
  • 适合需要轻量化情绪识别的智能交互与心理健康应用。

语音情绪识别需同时捕捉语言和副语言线索,在人机交互与心理健康监测中具有重要意义。近年来,大语言模型(LLM)展现出跨自然语言任务的能力,部分研究通过预训练主干网络与融合机制实现多模态结合。本文提出一种新方法:利用音频特征提取器获取音频特征,经可学习接口模块映射至LLM表示空间,并联合输入转换后的音频特征、文本转录及情绪预测任务提示。为高效适配,采用低秩适应(LoRA)实现参数高效的微调。在多个标准情绪识别基准上的实验表明,本模型性能仅次于少数领先方法,但参数量不足同类方法的一半,验证了其在语音情绪理解中融合多模态信息的有效性与显著计算效率优势。

原文摘要 · Abstract (English)

Emotion recognition from speech is a challenging task that requires capturing both linguistic and paralinguistic cues, with critical applications in human-computer interaction and mental health monitoring. Recent works have highlighted the ability of Large Language Models (LLMs) to perform tasks outside of the sole natural language area. In particular, recent approaches have investigated coupling LLMs with other data modalities by using pre-trained backbones and different fusion mechanisms. This work proposes a novel approach that fine-tunes an LLM with audio and text representations for emotion prediction. Our method first extracts audio features using an audio feature extractor, which are then mapped into the LLM's representation space via a learnable interfacing module. The LLM takes as input (1) the transformed audio features, (2) additional features in the form of natural language (e.g., the transcript), and (3) a textual prompt describing the emotion prediction task. To efficiently adapt the LLM to this multimodal task, we employ Low-Rank Adaptation (LoRA), enabling parameter-efficient fine-tuning. Experimental results on standard emotion recognition benchmarks demonstrate that our model outperforms all but one existing Speech-Text LLMs in the literature, while requiring less than half the parameters of competing approaches. This highlights our approach's effectiveness in integrating multi-modal inputs for speech-based emotion understanding while maintaining significant computational efficiency.

语音情绪大模型参数高效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。