通过分层概率变化提升生成内容真实性，无需额外训练。
Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Models
- 基于多层隐藏状态变化，逐令牌评估事实性。
- 在多个基准上显著降低幻觉率，保持问答准确率。
- 适合关注生成内容可信度的研究者与应用开发者。
尽管大语言模型具备强大能力，但仍常产生不准确或虚构的内容。本文将隐藏状态预测变化与输出事实性的关联拓展到逐令牌层面，提出无需额外训练的解码方法END（Cross-layer Entropy eNhanced Decoding）。END利用不同层间概率变化，量化每个候选词所需的事实知识，并调整最终预测分布以优先选择更真实的内容。在幻觉检测和问答基准上的实验表明，END显著提升了生成内容的真实性与信息量，同时保持了稳定的问答准确率。本工作也为理解模型内在知识与输出事实性之间的关系提供了新视角。
原文摘要 · Abstract (English)
Despite their impressive capacities, Large language models (LLMs) often struggle with the hallucination issue of generating inaccurate or fabricated content even when they possess correct knowledge. In this paper, we extend the exploration of the correlation between hidden-state prediction changes and output factuality into a deeper, token-wise level. Based on the insights , we propose cross-layer Entropy eNhanced Decoding (END), a decoding method that mitigates hallucinations without requiring extra training. END leverages inner probability changes across layers to individually quantify the factual knowledge required for each candidate token, and adjusts the final predicting distribution to prioritize tokens with higher factuality. Experiments on both hallucination and QA benchmarks demonstrate that END significantly enhances the truthfulness and informativeness of generated content while maintaining robust QA accuracy. Moreover, our work provides a deeper perspective on understanding the correlations between inherent knowledge and output factuality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。