研究大模型在说话时如何听用户讲话,找出两种路由策略的优劣。
How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue

- 将用户语音流直接注入模型或通过交叉注意力访问,两种不同处理方式。
- 直接注入提升问答准确率,但易受打断干扰导致内容混乱。
- 交叉注意力更抗干扰,适合复杂对话场景,尤其用户频繁插话时。
全双工语音对话要求模型在生成回应的同时持续监听用户输入,这对以单序列连贯性设计的大语言模型(LLM)构成挑战。本文探讨用户流如何被引入模型这一关键架构问题,将一个纯文本LLM扩展为统一的全双工语音对话系统,在同一训练框架下对比两种路由策略:(i) 通道融合,将用户流直接注入模型输入;(ii) 交叉注意力路由,将用户流作为外部记忆通过交叉注意力适配器访问。在语音问答和全双工交互基准上的实验显示明确权衡:通道融合在语义对齐上表现更强,问答性能更优;但在语义重叠场景(如用户打断)中更易受上下文污染——若模型未能及时停止,重叠用户流会干扰生成过程,导致语义不连贯。交叉注意力路由在问答任务中表现稍差,但能更好保持生成上下文,对失败模式更具鲁棒性。该研究确立了用户流路由是全双工对话的核心设计维度,并为语义整合与上下文鲁棒性之间的权衡提供了实用指导。我们提供演示页面供定性分析。
原文摘要 · Abstract (English)
Full-duplex spoken dialogue requires a model to keep listening while generating its own spoken response. This is challenging for large language models (LLMs), which are designed to extend a single coherent sequence and do not naturally support user input arriving during generation. We argue that how the user stream is routed into the LLM is therefore a key architectural question for full-duplex modeling. To study this question, we extend a text-only LLM into a unified full-duplex spoken dialogue system and compare two routing strategies under a shared training pipeline: (i) channel fusion, which injects the user stream directly into the LLM input, and (ii) cross-attention routing, which keeps the user stream as external memory accessed through cross-attention adapters. Experiments on spoken question answering and full-duplex interaction benchmarks reveal a clear tradeoff. Channel fusion yields stronger semantic grounding and consistently better question-answering performance. However, under semantically overlapping conditions such as user interruptions, it is more vulnerable to context corruption: if the model fails to stop in time, the overlapping user stream can interfere with ongoing generation and lead to semantically incoherent continuations. Cross-attention routing underperforms on question answering, but better preserves the LLM generation context and is more robust to this failure mode. These results establish user-stream routing as a central design axis in full-duplex spoken dialogue and offer practical guidance on the tradeoff between semantic integration and context robustness. We provide a demo page for qualitative inspection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。