通过跳过深层解码层减少大模型幻觉,提升输出可靠性。
Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping

- 动态跳过易产生幻觉的深层解码层,保留前序层信息。
- 在多个模型和基准上显著降低幻觉率,提升输出一致性。
- 适合追求高可靠性生成的场景,如医疗、法律问答。
大型语言模型在多种自然语言任务中表现优异,但其输出常出现与事实不符的幻觉内容。本文通过全面的层级分析发现,幻觉主要源自更深的解码层。为此提出一种新解码框架DeLask(Decoder Layer Skipping),通过梯度下降方向变化判断异常层:利用连续解码步骤间梯度的余弦相似性计算漂移值,当下降方向反转时识别出问题层。不直接丢弃这些层,而是部分聚合其隐藏状态与前序层,兼顾一致性与纠错能力。大量实验表明,DeLask在不同模型和基准上均能持续缓解幻觉,增强整体可靠性,是一种轻量且通用的解码优化方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved strong performance across diverse natural language tasks, yet their outputs often suffer from hallucinations -- content that is misaligned with factual information. In this work, we conduct a comprehensive layer-wise analysis of the decoding process and reveal that hallucinations tend to originate from deeper decoder layers. To address this issue, we introduce \textbf{DeLask} (\textbf{De}coder \textbf{La}yer \textbf{Sk}ipping), a novel decoding framework that dynamically skips layers prone to producing hallucinations. DeLask leverages the theoretical insight that the forward computation of an $L$-layer Transformer is conditionally equivalent to $L$ steps of gradient descent. We define a \emph{driftance value} by computing the cosine similarity between gradients derived from consecutive decoder steps, identifying problematic layers when the descent direction reverses. Rather than discarding such layers entirely, DeLask partially aggregates their hidden states with preceding layers, thereby preserving consistency while suppressing erroneous signals. Extensive experiments across diverse LLMs and benchmarks demonstrate that DeLask consistently mitigates hallucinations and enhances overall reliability, providing a lightweight and generalizable decoding framework for improving the robustness of large-scale language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。