用序列输出训练让语音识别同时标记说话人角色，一次解码搞定。
Joint ASR and Speaker Role Tagging with Serialized Output Training
- 用角色令牌增强Whisper，通过序列输出训练联合建模。
- 在双人对话数据集上，多说话人字错误率降低超10%。
- 适合需要角色感知的对话系统开发与部署。
自动语音识别系统在大规模预训练模型推动下取得了显著进展。然而，当前多数系统仅关注语音转写，忽略了说话人角色识别，而这一功能对对话式人工智能至关重要。本文研究了序列化输出训练（SOT）在联合语音识别与说话人角色标记中的应用。通过向Whisper模型添加角色特定令牌，并采用SOT进行微调，使模型能够在单次解码过程中生成带角色信息的转写结果。我们在两个真实对话数据集上将该方法与先前自监督基线方法进行了对比。结果表明，该方法在多说话人词错误率（WER）上实现超过10%的降低，验证了其作为统一角色感知语音转写模型的可行性。
原文摘要 · Abstract (English)
Automatic Speech Recognition systems have made significant progress with large-scale pre-trained models. However, most current systems focus solely on transcribing the speech without identifying speaker roles, a function that is critical for conversational AI. In this work, we investigate the use of serialized output training (SOT) for joint ASR and speaker role tagging. By augmenting Whisper with role-specific tokens and fine-tuning it with SOT, we enable the model to generate role-aware transcriptions in a single decoding pass. We compare the SOT approach against a self-supervised previous baseline method on two real-world conversational datasets. Our findings show that this approach achieves more than 10% reduction in multi-talker WER, demonstrating its feasibility as a unified model for speaker-role aware speech transcription.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。