用说话人标注的Whisper模型实现多说话人语音联合识别
Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition
- 将说话人嵌入融合到Whisper编码器，通过共享解码器联合建模
- 在LibriMix数据集上优于现有SOT方法和DiCoW模型
- 适合需要高精度多说话人语音识别的应用场景
我们提出一种基于说话人属性（SA）的Whisper模型，用于多说话人语音识别，结合目标说话人建模与序列化输出训练（SOT）。该方法利用对话分离条件的Whisper（DiCoW）编码器提取目标说话人嵌入，将其合并为单一表示并输入共享解码器。模型可将重叠语音转录为带说话人标签和时间戳的序列化输出。与仅对每位说话人单独解码的DiCoW等目标说话人语音识别系统不同，本方法采用联合解码，使解码器能同时依赖所有说话人的上下文信息。实验表明，该模型在多说话人混合语料（如LibriMix）上的表现优于现有SOT方法，并超越DiCoW。
原文摘要 · Abstract (English)
We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。