让语音对话模型实时查知识,回答更准还不卡顿
MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

- 用异步检索机制,在说话间隙悄悄查外部资料
- 事实性接近非实时模型,延迟仍保持对话自然流畅
- 插件式设计支持换知识源,数学推理也靠谱
语音到语音语言模型最近兴起,以提升对话AI的自然度。全双工模型因其实时互动能力脱颖而出,可处理停顿、打断和回应信号。然而,提升其事实准确性仍是开放挑战。单纯增大模型规模虽能改善,但会使实时推理成本过高。本文提出MoshiRAG,一种模块化方法:结合紧凑的全双工接口与选择性检索,接入更强大的知识源。其异步框架可识别需知识的提问,并在响应生成期间完成外部信息检索。利用响应起始与核心信息送达间的自然时间间隙,检索可在不破坏对话流畅性的前提下完成。实验表明,MoshiRAG在事实性上达到当前公开最佳非全双工模型水平,同时保留全双工系统的交互特性。其灵活设计支持无需重训练即可更换检索方式,并在跨领域数学推理任务中表现优异。
原文摘要 · Abstract (English)
Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose MoshiRAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, MoshiRAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。