arXiv:2606.07547cs.CLcs.AI2026-06

让语音模型先写后说,实现实时文本输出与口语交互并行。

Liberating LLM Capabilities in Full-Duplex Speech Models

论文配图:Liberating LLM Capabilities in Full-Duplex Speech Models
图 1 · 摘自论文原文
  • 采用文本优先的三通道架构,语音输入、文本输出、语音回复同步进行。
  • 在多个评测中表现优异,写作与说话一致性达92.6%,优于内部对照组。
  • 无需修改模型结构,适合需要实时可见推理过程的交互场景。

基于语音的大语言模型通常仅限于口头回复,限制了其在需持续、结构化和可检查中间输出的任务中发挥文本原生能力,如代码生成、结构化分析和多步推理。现有工作虽改善了语音推理或全双工对话,但仍将文本视为隐藏中间状态或次要模态。本文提出听-写-说(LWS)范式,通过单一自回归大模型在共享因果注意力上下文中,持续接收用户音频,以可见自由文本作为主要输出,并并行生成实时语音回应。该行为由一个令牌模式实现,无需架构改动,通过两阶段数据管道合成与输入时间线一致的每秒认知标注进行训练。实验表明,LWS在Full-Duplex-Bench上表现良好,VoiceBench AlpacaEval得分为4.72,写作与说话一致性达92.6%,并在URO-Bench上持续优于内部消融实验。结果表明,可见书写可作为语音交互的一等输出通道,且不牺牲实时响应性。代码与数据集已公开于项目页:https://royalzhang.com/project/lws-page/。

原文摘要 · Abstract (English)

Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs. Existing work improves spoken reasoning or full-duplex turn-taking, but still treats text as a hidden intermediate state or a subordinate modality rather than a first-class output channel. We propose Listen-Write-Speak (LWS), a text-first tri-channel paradigm in which a single autoregressive LLM continuously listens to user audio, writes visible free-form text as its primary output, and speaks a realtime oral response in parallel under a shared causal attention context. This behavior is implemented entirely through a Token Schema, requiring no architectural modifications, and learned via a two-stage data pipeline that synthesizes per-second cognitive annotations consistent with the revealed input timeline. Empirically, LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. These results suggest that visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime responsiveness. The code and dataset are available on the project page: https://royalzhang.com/project/lws-page/.

语音交互文本生成大模型全双工

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。