让语音模型直接输出单词时间戳，提升字幕与检索精度
Word Level Timestamp Generation for Automatic Speech Recognition and Translation
- 用教师模型生成时间戳，训练模型直接预测每个词的起止时间
- 四语言时间戳误差20~120毫秒，准确率与召回率80%~90%
- 可直接用于语音翻译，时间戳误差约200毫秒，性能损失小
我们提出一种数据驱动方法，使Canary模型具备词级时间戳预测能力。精确的时间戳对语音内容检索、定时字幕等下游任务至关重要。传统混合系统与端到端模型需依赖外部对齐模块，而本方法无需额外对齐机制。通过NeMo强制对齐器（NFA）作为教师模型生成词级时间戳，训练Canary模型直接预测时间戳。引入新特殊标记<|timestamp|>，使模型能输出每个词的起始与结束时间。实验显示，该方法在四种语言上时间戳预测误差为20至120毫秒，精度与召回率均达80%至90%，且词错误率（WER）仅轻微下降。此外，该系统扩展至自动语音翻译（AST）任务，时间戳误差约为200毫秒。
原文摘要 · Abstract (English)
We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model. Accurate timestamp information is crucial for a variety of downstream tasks such as speech content retrieval and timed subtitles. While traditional hybrid systems and end-to-end (E2E) models may employ external modules for timestamp prediction, our approach eliminates the need for separate alignment mechanisms. By leveraging the NeMo Forced Aligner (NFA) as a teacher model, we generate word-level timestamps and train the Canary model to predict timestamps directly. We introduce a new <|timestamp|> token, enabling the Canary model to predict start and end timestamps for each word. Our method demonstrates precision and recall rates between 80% and 90%, with timestamp prediction errors ranging from 20 to 120 ms across four languages, with minimal WER degradation. Additionally, we extend our system to automatic speech translation (AST) tasks, achieving timestamp prediction errors around 200 milliseconds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。