让大模型理解说话方式和环境,提升语音交互的准确与高效
Scalable Context Orchestration for Serving LLMs Over Voice

- 显式建模语速、噪声等语音上下文,动态生成响应指令
- 在丢包条件下中断率从46%降至0.9%,语速对齐误差降低52.4%
- 长对话中每轮成本降为1/24.9,适合高并发语音应用
语音人工智能应用因大语言模型的发展而日益普及,实现了更自然、易用的语音交互。但服务此类应用需兼顾用户所说内容、说话方式(如语速)以及音频采集与传输条件(如背景噪声、丢包)。现有LLM系统将对话上下文表示为扁平递增的消息序列,使语音相关上下文隐含于音频中,导致响应与用户偏好不匹配,环境恶劣时交互质量下降,长时间会话成本高昂。本文提出llmovoice,一种上下文管理中间件,显式建模语音上下文并协调其使用。每轮对话中,llmovoice从当前输入、相关历史及显式的副语言与环境状态构建有界语音上下文,再由推理模型在此上下文中生成运行时指令以指导系统响应。在真实语音应用和基准测试中评估表明,llmovoice将语速对齐误差降低52.4%,在丢包条件下误打断率从46.0%降至0.9%,模型使用成本减少79.2%。长期会话中,每轮成本最高降低24.9倍,同时保留高达98.7%的基线回答质量。
原文摘要 · Abstract (English)
Voice AI applications are gaining popularity as advances in large language models (LLMs) enable more natural and accessible spoken interactions. Serving these applications requires accounting not only for what users say, but also for how they speak (e.g., speaking rate) and the conditions under which their audio is captured and transmitted (e.g., background noise and packet loss). However, existing LLM systems represent conversation context as a flat, growing sequence of messages, leaving voice-specific context implicit in the audio. As a result, they can generate responses that are poorly aligned with user preferences, degrade interaction quality under adverse environmental conditions, and incur high costs over long voice sessions. We present llmovoice, a context-management middleware that explicitly models voice context and orchestrates its use. At each turn, llmovoice constructs a bounded voice context from the current user input, relevant interaction history, and explicit paralinguistic and environmental states. It then uses the serving LLM to reason over this context and generate runtime directives that guide how the system responds. We evaluate llmovoice on real-world voice applications and benchmarks. It reduces speaking-rate alignment error by 52.4%, lowers the false-interruption rate from 46.0% to 0.9% under packet loss, and reduces model usage cost by 79.2%. For long sessions, llmovoice reduces per-turn cost by up to 24.9 times while retaining up to 98.7% of baseline answer quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。