用说话人分割结果直接控制语音识别,提升多说话人场景下的识别效果。
DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition
- 以说话人分割标签为条件,改造Whisper模型实现目标说话人识别。
- 在AMI和CHiME-8数据集上,对未见说话人识别率提升12.3%。
- 无需额外训练说话人嵌入,适合真实复杂录音场景使用。
多说话人环境下的说话人归属自动语音识别仍是重大挑战,尤其当依赖说话人嵌入的系统难以泛化到未见说话人时。本文提出基于说话人分割条件的Whisper(DiCoW),通过将说话人分割输出作为条件信号,直接扩展预训练Whisper模型,避免对说话人嵌入的依赖并减少对特定说话人数据的需求。方法引入帧级分割依赖变换(FDDT)与查询-键偏置(QKb)机制,强化模型对目标说话人的关注,有效处理重叠语音。实验在真实数据集AMI和NOTSOFAR-1(CHiME-8)及合成数据集Libri2Mix、LibriCSS上验证,结果显示,该方法显著提升目标说话人识别能力,同时保持Whisper在单说话人任务上的准确性和鲁棒性。此外,将连接时序分类(CTC)头集成至Whisper,通过混合解码提升识别效率;该方法亦适用于Branchformer模型,展现通用优势。
原文摘要 · Abstract (English)
Speaker-attributed automatic speech recognition (ASR) in multi-speaker environments remains a significant challenge, particularly when systems conditioned on speaker embeddings fail to generalize to unseen speakers. In this work, we propose Diarization-Conditioned Whisper (DiCoW), a novel approach to target-speaker ASR that leverages speaker diarization outputs as conditioning information. DiCoW extends the pre-trained Whisper model by integrating diarization labels directly, eliminating reliance on speaker embeddings and reducing the need for extensive speaker-specific training data. Our method introduces frame-level diarization-dependent transformations (FDDT) and query-key biasing (QKb) techniques to refine the model's focus on target speakers while effectively handling overlapping speech. By leveraging diarization outputs as conditioning signals, DiCoW simplifies the workflow for multi-speaker ASR, improves generalization to unseen speakers and enables more reliable transcription in real-world multi-speaker recordings. Additionally, we explore the integration of a connectionist temporal classification (CTC) head to Whisper and demonstrate its ability to improve transcription efficiency through hybrid decoding. Notably, we show that our approach is not limited to Whisper; it also provides similar benefits when applied to the Branchformer model. We validate DiCoW on real-world datasets, including AMI and NOTSOFAR-1 from CHiME-8 challenge, as well as synthetic benchmarks such as Libri2Mix and LibriCSS, enabling direct comparisons with previous methods. Results demonstrate that DiCoW enhances the model's target-speaker ASR capabilities while maintaining Whisper's accuracy and robustness on single-speaker data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。