用说话人分段信息让单说话人模型变目标说话人识别模型
Target Speaker ASR with Whisper
- 用帧级说话人分段结果做条件，学习说话人间相对差异
- 仅加一个偏置项就使Whisper实现目标说话人识别，准确率提升12.9%
- 支持逐说话人生成转录，适合多说话人语音场景
我们提出一种新方法，使大型单说话人语音识别模型（如Whisper）可用于目标说话人语音识别。核心思想是：通过在帧级说话人分段输出上施加条件，学习说话人间的相对差异，比直接建模所有说话人嵌入空间更高效。实验发现，在首个Transformer块前为每类分段输出添加单一偏置项，即可将单说话人模型转化为目标说话人模型。该方法还支持按分段顺序为每位说话人生成转录。在NOTSOFAR-1数据集上，相比基线语音分离与分段级联方法，绝对降低12.9%的ORC-WER。
原文摘要 · Abstract (English)
We propose a novel approach to enable the use of large, single-speaker ASR models, such as Whisper, for target speaker ASR. The key claim of this method is that it is much easier to model relative differences among speakers by learning to condition on frame-level diarization outputs than to learn the space of all speaker embeddings. We find that adding even a single bias term per diarization output type before the first transformer block can transform single-speaker ASR models into target-speaker ASR models. Our approach also supports speaker-attributed ASR by sequentially generating transcripts for each speaker in a diarization output. This simplified method outperforms baseline speech separation and diarization cascade by 12.9 % absolute ORC-WER on the NOTSOFAR-1 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。