通过说话人分离条件控制，让语音大模型在远场多人对话中更准确地识别说话人。
Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning
- 用说话人分离掩码控制声学编码器，冻结解码器避免遗忘。
- 在多个数据集上比Gemini等模型低29%的说话人误标率。
- 无需微调就能媲美远场内容理解，微调后超越近场模型表现。
我们提出基于说话人分离条件的语音语言模型（SLMs），用于处理远场多人音频。不同于通过序列输出训练适配解码器（易导致灾难性遗忘），我们通过说话人分离掩码来条件化声学编码器，以提取目标说话人表征，同时保持解码器不变。我们构建了Dixtral模型，将说话人分离条件化的Whisper（DiCoW）编码器集成到Voxtral SLM中。在AMI、NOTSOFAR-1、LibriSpeechMix和Mixer6数据集上，Dixtral在说话人归属转录任务中分别优于Gemini 3.0 Flash、VibeVoice和Voxtral Mini Transcribe V2，绝对cpWER降低29.0%、19.8%和16.0%。在一个全新的长时多说话人问答基准上，零样本Dixtral在远场内容理解上达到与Gemini相当的水平；经微调后，在所有任务上均超越采用近场音频的Gemini和Voxtral。
原文摘要 · Abstract (English)
We propose diarization-conditioned spoken language models (SLMs), a strategy for extending SLMs to far-field multi-talker audio. Rather than adapting the decoder via Serialized Output Training, which risks catastrophic forgetting, we condition the acoustic encoder on diarization masks to extract target-speaker representations, keeping the decoder frozen. We instantiate this as Dixtral, integrating a Diarization Conditioned Whisper (DiCoW) encoder into the Voxtral SLM. On AMI, NOTSOFAR-1, LibriSpeechMix, and Mixer6, Dixtral outperforms Gemini 3.0 Flash, VibeVoice, and Voxtral Mini Transcribe V2 on speaker-attributed transcription by 29.0%, 19.8%, and 16.0% absolute cpWER respectively. On a novel long-form multi-speaker QA benchmark, zero-shot Dixtral matches Gemini on far-field content understanding, and when fine-tuned surpasses both Gemini and Voxtral operating on close-talk across all tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。