通过投影抑制Whisper在无声输入时的幻觉输出,无需重新训练。
Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection
- 用非语音数据提取幻觉子空间,推理时将解码器状态投影远离该空间。
- 始终开启模式下幻觉率从31.31%降至2.44%,降低92.21%。
- 门控模式可平衡幻觉抑制与真实语音误拒,适合实际部署场景。
Whisper是广泛使用的自动语音识别基础模型,但其生成式解码器在无语音或弱语音输入时会产生流畅的幻觉转录。本文提出一种无需训练、仅在推理阶段生效的方法:对解码器激活进行低秩投影。通过非语音校准数据估计出一个紧凑的幻觉相关子空间,并在推理时将解码器隐藏状态投影远离该子空间。评估两种变体:始终开启模式对所有输入应用投影,门控模式仅在Whisper预测输入可能为非语音时启用。在非语音基准测试中,始终开启模式将平均幻觉率(HR)从31.31%降至2.44%,相对减少92.21%;门控模式将幻觉率降至3.74%,相对减少88.05%,同时降低对真实语音的误拒。在LibriSpeech上,门控模式使绝对词错误率(WER)增加0.33–4.39个百分点,不同模型和划分设置下的误拒率(FRR)为0.41%–9.97%。结果表明,低秩激活投影可显著抑制Whisper幻觉,且无需重训练,同时提供幻觉抑制与识别性能间的可控权衡。
原文摘要 · Abstract (English)
Whisper is a widely used foundation model for automatic speech recognition (ASR), but its generative decoder can produce fluent hallucinated transcripts for inputs containing little or no speech. We propose a training-free, inference-time method to reduce these hallucinations using low-rank projection of decoder activations. A compact hallucination-associated subspace is estimated from non-speech calibration data, and decoder hidden states are projected away from this subspace during inference. We evaluate two variants: always-on, which applies projection to all inputs, and gated, which applies it only when Whisper predicts that an input is likely non-speech. Across non-speech benchmarks, always-on projection reduces average hallucination rate (HR) from 31.31% to 2.44%, a 92.21% relative reduction, while gated projection reduces HR to 3.74%, an 88.05% relative reduction, with lower false rejection of genuine speech. On LibriSpeech, gated projection increases absolute word error rate (WER) by 0.33-4.39 percentage points and yields false-rejection rates (FRR) of 0.41--9.97% across model and split settings. These results show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining, while providing a controllable trade-off between hallucination suppression and speech recognition performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。