用排序损失解决语音识别中说话人标签的混乱问题，提升多人对话转写准确率。
Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
- 引入排序损失（Sort Loss）替代或补充传统排列不变损失
- 在多说话人语音转录中实现更精准的说话人标签对齐
- 适合需要说话人区分的语音识别与大模型应用
Sortformer 是一种基于编码器的说话人分离模型，用于监督语音到文本模型中的说话人标签。它不依赖单一的排列不变损失（PIL），而是引入排序损失（Sort Loss）来解决排列问题，可独立使用或与 PIL 结合。此外，我们提出一种简化的多说话人语音到文本架构，利用 Sortformer 通过正弦核函数将说话人标签嵌入编码器，通过排序目标有效连接时间戳与输出标记，实现对转录结果中说话人标签的监督。实验表明，排序损失能提升说话人分离性能，且结合 Sortformer 的说话人监督可显著提高多说话人转录准确性。我们预期该模型与架构将推动说话人标签能力无缝集成至基础语音到文本系统及多模态大语言模型（LLMs），为语音感知任务提供易用高效的增强机制。代码与训练模型已通过 NVIDIA NeMo 框架开源。
原文摘要 · Abstract (English)
Sortformer is an encoder-based speaker diarization model designed for supervising speaker tagging in speech-to-text models. Instead of relying solely on permutation invariant loss (PIL), Sortformer introduces Sort Loss to resolve the permutation problem, either independently or in tandem with PIL. In addition, we propose a streamlined multi-speaker speech-to-text architecture that leverages Sortformer for speaker supervision, embedding speaker labels into the encoder using sinusoidal kernel functions. This design addresses the speaker permutation problem through sorted objectives, effectively bridging timestamps and tokens to supervise speaker labels in the output transcriptions. Experiments demonstrate that Sort Loss can boost speaker diarization performance, and incorporating the speaker supervision from Sortformer improves multi-speaker transcription accuracy. We anticipate that the proposed Sortformer and multi-speaker architecture will enable the seamless integration of speaker tagging capabilities into foundational speech-to-text systems and multimodal large language models (LLMs), offering an easily adoptable and user-friendly mechanism to enhance their versatility and performance in speaker-aware tasks. The code and trained models are made publicly available through the NVIDIA NeMo Framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。