无需训练，利用Whisper内部注意力机制实现高精度词级时间对齐
Whisper Has an Internal Word Aligner
- 通过筛选特定注意力头并用字符作为输入，实现无监督词对齐
- 在20-100毫秒严格容忍度下，精度优于已有方法
- 发现字符输入比词片段更利于生成精细对齐，适合语音分析研究者
目前对强自动语音识别器（特别是Whisper）进行精确词级时间戳提取的兴趣日益增长。现有方法要么需要额外训练，要么竞争力不足。先前研究的评估标准也较为宽松，通常容忍度超过200毫秒。本文发现Whisper中某些注意力头能捕捉到准确的词对齐，且与不具此能力的头有明显区别。此外，使用字符而非词片可产生更精细、更准确的对齐结果。基于此，我们提出一种无监督方法：通过过滤注意力头，并在教师强制（teacher forcing）下使用字符输入来提取词对齐。该方法无需训练，且在20至100毫秒的严格容忍度下，对齐精度高于以往工作。
原文摘要 · Abstract (English)
There is an increasing interest in obtaining accurate word-level timestamps from strong automatic speech recognizers, in particular Whisper. Existing approaches either require additional training or are simply not competitive. The evaluation in prior work is also relatively loose, typically using a tolerance of more than 200 ms. In this work, we discover attention heads in Whisper that capture accurate word alignments and are distinctively different from those that do not. Moreover, we find that using characters produces finer and more accurate alignments than using wordpieces. Based on these findings, we propose an unsupervised approach to extracting word alignments by filtering attention heads while teacher forcing Whisper with characters. Our approach not only does not require training but also produces word alignments that are more accurate than prior work under a stricter tolerance between 20 ms and 100 ms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。