通过精调特定注意力头，显著降低Whisper在非语音段的幻觉问题。
Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down
- 定位并重训导致幻觉的3个关键注意力头。
- 非语音幻觉减少超80%，语音识别误差仅上升不到0.1%。
- 无需预处理或后处理，适合工业级语音识别场景。
OpenAI的Whisper在自动语音识别中表现卓越，但其在非语音段常出现幻觉，限制了在复杂工业环境中的应用。本文提出一种新方法，在不使用任何预处理或后处理的前提下，通过逐头掩码分析Whisper-large-v3解码器中各自注意力头对幻觉的贡献。实验发现，仅3个头就导致了UrbanSound数据集上超过75%的幻觉。随后，我们用非语音数据对这三个‘疯狂’头进行微调，得到最佳模型Calm-Whisper。结果表明，该模型在非语音幻觉上减少超80%，同时在LibriSpeech test-clean和test-other上的词错误率（WER）仅增加不足0.1%。
原文摘要 · Abstract (English)
OpenAI's Whisper has achieved significant success in Automatic Speech Recognition. However, it has consistently been found to exhibit hallucination issues, particularly in non-speech segments, which limits its broader application in complex industrial settings. In this paper, we introduce a novel method to reduce Whisper's hallucination on non-speech segments without using any pre- or post-possessing techniques. Specifically, we benchmark the contribution of each self-attentional head in the Whisper-large-v3 decoder to the hallucination problem by performing a head-wise mask. Our findings reveal that only 3 of the 20 heads account for over 75% of the hallucinations on the UrbanSound dataset. We then fine-tune these three crazy heads using a collection of non-speech data. The results show that our best fine-tuned model, namely Calm-Whisper, achieves over 80% reduction in non-speech hallucination with only less than 0.1% WER degradation on LibriSpeech test-clean and test-other.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。