首个统一音频前端大模型,让语音交互更自然实时。
UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction

- 将语音活动检测等任务统一为序列生成,端到端处理
- 实测响应延迟降低,打断准确率显著提升
- 适合需要实时交互的智能助手、会议系统等场景
全双工语音交互是人类最自然的沟通方式,正推动人工智能向更类人对话系统发展。传统级联式语音处理流程存在延迟累积、信息丢失和错误传播等问题。近期研究虽聚焦于统一语音理解与生成的端到端大模型(如GPT-4o),但多数仍为半双工,依赖独立的任务专用前端组件(如语音活动检测VAD、说话人转换检测TD)。在开发语音助手过程中,我们发现优化前端同样关键。为此,提出首个面向全双工系统的统一音频前端大模型(UAF)。该模型将VAD、TD、说话人识别(SR)、自动语音识别(ASR)及问答(QA)等多样任务重构为单一自回归序列预测问题。输入为固定时长音频块(如600毫秒),利用参考音频提示锚定目标说话人,逐帧生成编码语义内容与系统状态控制(如打断信号)的离散标记。实验表明,该模型在多个前端任务上达到领先性能,并显著提升真实交互场景下的响应延迟与打断准确性。
原文摘要 · Abstract (English)
Full-duplex speech interaction, as the most natural and intuitive mode of human communication, is driving artificial intelligence toward more human-like conversational systems. Traditional cascaded speech processing pipelines suffer from critical limitations, including accumulated latency, information loss, and error propagation across modules. To address these issues, recent efforts focus on the end-to-end audio large language models (LLMs) like GPT-4o, which primarily unify speech understanding and generation task. However, most of these models are inherently half-duplex, and rely on a suite of separate, task-specific front-end components, such as voice activity detection (VAD) and turn-taking detection (TD). In our development of speech assistant, we observed that optimizing the speech front-end is equally crucial as advancing the back-end unified model for achieving seamless, responsive interactions. To bridge this gap, we propose the first unified audio front-end LLM (UAF) tailored for full-duplex speech systems. Our model reformulates diverse audio front-end tasks into a single auto-regressive sequence prediction problem, including VAD, TD, speaker recognition (SR), automatic speech recognition (ASR) and question answer (QA). It takes streaming fixed-duration audio chunk (e.g., 600 ms) as input, leverages a reference audio prompt to anchor the target speaker at the beginning, and regressively generates discrete tokens encoding both semantic content and system-level state controls (e.g., interruption signals). Experiments demonstrate that our model achieves leading performance across multiple audio front-end tasks and significantly enhances response latency and interruption accuracy in real-world interaction scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。