通过分析模型内部状态，提升语音识别幻觉检测效果
From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection
- 结合文本特征与模型解码层状态进行幻觉检测
- 无需参考文本即可实现最高准确率，关键在中间层表征
- 适合关注语音识别可靠性与模型可解释性的研究者
语音识别模型的幻觉现象——即生成流畅但与音频无关的转录结果——会降低系统性能并带来下游应用风险。本文针对真实语音的人工标注数据，在三种范式下研究 Whisper large v3 的幻觉检测：基于文本的、基于大语言模型的和对解码器内部状态的探查。利用文本评估指标的文本分类器虽具高召回率，但在无参考转录时性能下降；基于大语言模型的方法通过领域特定提示增强精度，但仍不如轻量级文本方法。探查解码器表示发现，幻觉特征编码于多个中间解码层中，且无需真实参考文本即可实现最优性能。通过融合文本与内部状态输出的元分类器，达到最佳整体检测效果。
原文摘要 · Abstract (English)
Hallucinations of ASR models - fluent transcriptions with no basis in audio - degrade system performance and pose risks in downstream applications. Robust detection of such errors remains a challenge. This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data across three paradigms: text-based, LLM-based, and internal decoder state probing. Text classifiers utilizing metrics for text evaluation achieve high recall but degrade without reference transcripts. LLM-based detection improves precision with domain-specific prompt conditioning, yet remains less competitive than the lightweight text-based methods. Probing Whisper's decoder representations, without a ground-truth reference, yields the strongest performance, revealing that hallucination traits are encoded across intermediate decoding layers. A late-fusion meta-classifier combining text and internal-state outputs achieves the best overall detection performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。