不微调大模型，让其理解语音的语气情感
Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech
- 用语音编码器生成融合语义与语气的嵌入
- 零微调下响应质量优于多个基线
- 适合需要情感理解的对话系统
本文研究了大型语言模型（LLM）在不微调权重的情况下，理解语音非语言特征的能力。我们构建了一个端到端系统，采用语音编码器将语音转化为词元嵌入，使模型对带有情感色彩的语音提示的响应，与对语义匹配但包含说话风格文本提示的响应保持一致。该框架使编码器生成的嵌入同时包含语言和非语言信息，并有效传递给完全冻结的LLM。据我们所知，这是首个在通用交互场景中探索如何让冻结的LLM理解语音情感等非语言内容的工作。实验表明，该系统对表达性语音提示的响应质量更高、更具同理心，优于多个基线方法。
原文摘要 · Abstract (English)
This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt that has also been conditioned on the user's speaking style. This framework enables the encoder to generate tokens that capture both linguistic and paralinguistic information and effectively convey them to the LLM, even when the LLM's weights remain completely frozen. To the best of our knowledge, our work is the first to explore how to induce a frozen LLM to understand more than just linguistic content from speech inputs in a general interaction setting. Experiments demonstrate that our system is able to produce higher quality and more empathetic responses to expressive speech prompts compared to several baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。