LUCY让语音助手能懂情绪、自然回应并调用工具，更像真人对话伙伴。
LUCY: Linguistic Understanding and Control Yielding Early Stage of Her
- 端到端模型结合语言与语气线索，实现情绪感知与控制。
- 在情感响应上优于同类模型，且生成语句更自然流畅。
- 可调用外部工具回答知识外问题，适合需要实时交互的场景。
电影《她》中的萨曼莎是一位能理解人类言语中语言与副语言信息的高级语音代理，可实时生成自然、有信息量且敏感于情感细微差别的回应。为推动端到端（E2E）语音系统向更智能语音代理迈进，我们提出LUCY，一种能（1）感知并响应用户情绪，（2）以简洁自然风格生成回复，（3）调用外部工具解答实时问题的E2E语音模型。实验表明，相比同类模型，LUCY在情绪控制方面表现更优，能根据语言指令和副语言情绪线索生成相应反应；同时，在外部语言模型评估中，其回复风格更自然，且未显著牺牲通用问答性能。此外，LUCY可通过函数调用回答超出自身知识范围的问题。
原文摘要 · Abstract (English)
The film Her features Samantha, a sophisticated AI audio agent who is capable of understanding both linguistic and paralinguistic information in human speech and delivering real-time responses that are natural, informative and sensitive to emotional subtleties. Moving one step toward more sophisticated audio agent from recent advancement in end-to-end (E2E) speech systems, we propose LUCY, a E2E speech model that (1) senses and responds to user's emotion, (2) deliver responses in a succinct and natural style, and (3) use external tool to answer real-time inquiries. Experiment results show that LUCY is better at emotion control than peer models, generating emotional responses based on linguistic emotional instructions and responding to paralinguistic emotional cues. Lucy is also able to generate responses in a more natural style, as judged by external language models, without sacrificing much performance on general question answering. Finally, LUCY can leverage function calls to answer questions that are out of its knowledge scope.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。