通过自适应门控与值校准,减少大模型生成幻觉。
HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models
- 根据输入动态调整注意力头权重,实现实例级软重加权。
- 结合值向量幅度校准注意力,更准确反映词元贡献度。
- 无需微调,单次前向传播即可部署,适合实际应用。
大语言模型在检索增强或长文本生成中常产生幻觉,即使存在相关证据。原因在于:注意力头重要性被当作与输入无关,且原始注意力权重未能真实反映词元的贡献。我们提出HAVE(Head-Adaptive Gating and ValuE Calibration),一种无参数解码框架,直接解决上述问题。HAVE引入头自适应门控,对注意力头进行实例级软重加权;并引入值校准,将值向量的幅值融入注意力,以近似写回贡献。两者共同构建与模型更新对齐的词元级证据,并通过轻量级不确定性缩放策略融合到语言模型分布中。HAVE无需微调,仅需一次前向传播,高效且可广泛适用。在多个问答基准和大模型家族上的实验表明,HAVE持续降低幻觉率,优于强基线如DAGCD,且开销微小。该框架透明、可复现,可轻松集成至现成大模型,推动真实场景中的可信生成。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often produce hallucinations in retrieval-augmented or long-context generation, even when relevant evidence is present. This stems from two issues: head importance is treated as input-agnostic, and raw attention weights poorly reflect each token's true contribution. We present HAVE (Head-Adaptive Gating and ValuE Calibration), a parameter-free decoding framework that directly addresses both challenges. HAVE introduces head-adaptive gating, which performs instance-level soft reweighing of attention heads, and value calibration, which augments attention with the magnitude of value vectors to approximate write-back contribution. Together, these modules construct token-level evidence aligned with model updates and fuse it with the LM distribution through a lightweight uncertainty-scaled policy. HAVE requires no finetuning and operates in a single forward pass, making it efficient and broadly applicable. Experiments across multiple QA benchmarks and LLM families demonstrate that HAVE consistently reduces hallucinations and outperforms strong baselines, including DAGCD, with modest overhead. The framework is transparent, reproducible, and readily integrates with off-the-shelf LLMs, advancing trustworthy generation in real-world settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。