提升多人语音识别的实时与离线表现，兼顾速度与准确率。
Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarios
- 用单通道语音分离技术改进端到端系统，在重叠语音中提升识别效果。
- 采用双模型架构：流式用Conformer Transducer，离线用序列到序列模型。
- 提出分段SOT方法，让多人转录更清晰，适合离线场景使用。
我们将序列化输出训练（SOT）框架扩展至满足流式与离线自动语音识别（ASR）的实际需求。方法重点在平衡延迟与准确性，以适应实时字幕与摘要生成任务。提出三项关键改进：（1）在高度重叠语音场景下，采用单通道语音分离（CSS）作为前端，结合端到端（E2E）系统，挑战传统端到端与级联架构的优劣观念，显著提升识别准确率；（2）部署双模型策略——流式使用Conformer Transducer，离线使用序列到序列模型，或采用基于级联编码器的两阶段模型；（3）探索分段SOT（segSOT），更适配离线场景，并增强多人转录结果的可读性。
原文摘要 · Abstract (English)
We extend the frameworks of Serialized Output Training (SOT) to address practical needs of both streaming and offline automatic speech recognition (ASR) applications. Our approach focuses on balancing latency and accuracy, catering to real-time captioning and summarization requirements. We propose several key improvements: (1) Leveraging Continuous Speech Separation (CSS) single-channel front-end with end-to-end (E2E) systems for highly overlapping scenarios, challenging the conventional wisdom of E2E versus cascaded setups. The CSS framework improves the accuracy of the ASR system by separating overlapped speech from multiple speakers. (2) Implementing dual models -- Conformer Transducer for streaming and Sequence-to-Sequence for offline -- or alternatively, a two-pass model based on cascaded encoders. (3) Exploring segment-based SOT (segSOT) which is better suited for offline scenarios while also enhancing readability of multi-talker transcriptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。