arXiv:2505.17060cs.CLcs.AI2025-05NeurIPS被引 33

首个无需编码器的独立语音大模型,实现自然双工对话。

SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation

  • 用动态思维机制让模型自主切换听与说状态,无需外部模块。
  • 在多项评测中性能比现有开源模型高30%以上,用更少数据达到顶尖水平。
  • 擅长抢话、回声消除等复杂场景,适合构建真实语音助手。

为实现流畅自然的人机语音交互,现有全双工对话系统常采用包含语音活动检测器、打断器、对话状态预测器或多模型的模块化架构,但存在模块误差累积问题,且难以应对上下文依赖抢话和回声消除等挑战。近期方法如Moshi通过将音频编码器注入单个大模型的词元空间简化流程,但仍在语音模态上表现明显下降。本文提出SALMONN-omni,首个无需在词元空间注入音频编码器的独立全双工语音大模型。其核心是大模型主干中的动态思维机制,使模型能自主学习何时切换至说话或倾听状态。在广泛使用的语音问答与开放域对话基准测试中,SALMONN-omni相比现有开源全双工模型至少提升30%相对性能,并在半双工与轮次制系统中表现出高度竞争力,尽管训练数据显著减少。此外,该模型在复杂对话场景(包括换言、反馈语、回声消除、上下文依赖抢话)中表现优异,强化学习进一步提升了效果。演示对话见GitHub仓库:https://github.com/bytedance/SALMONN。

原文摘要 · Abstract (English)

In order to enable fluid and natural human-machine speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such as voice activity detectors, interrupters, conversation state predictors, or multiple LLMs. These systems, however, suffer from error accumulation across modules and struggle with key challenges such as context-dependent barge-in and echo cancellation. Recent approaches, most notably Moshi, simplify the pipeline by injecting audio codecs into the token space of a single LLM. However, such methods still incur significant performance degradation when operating on the speech rather than text modality. In this paper, we introduce SALMONN-omni, the first single, standalone full-duplex speech LLM that operates without audio codecs in its token space. It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when to transition between speaking and listening states. Experiments on widely used benchmarks for spoken question answering and open-domain dialogue show that SALMONN-omni achieves at least 30\% relative performance improvement over existing open-source full-duplex models and performs highly competitively to half-duplex and turn-based systems, despite using substantially less training data. Moreover, SALMONN-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling, echo cancellation and context-dependent barge-in, with further improvements achieved through reinforcement learning. Some demo conversations between user and SALMONN-omni are provided in the following repository https://github.com/bytedance/SALMONN.

语音大模型全双工对话动态决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。