通过提前层插值提升大模型事实性，无需训练即可减少幻觉。
Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation
- 在模型早期层间插入数学插值层，扩展信息处理深度。
- 在4个数据集上显著降低幻觉率，多数情况优于现有方法。
- 无需训练、可即插即用，适合希望提升输出准确性的开发者。
大语言模型在文本理解和生成方面表现出色，但其产生的事实不一致（即‘幻觉’）仍是关键挑战。现有方法如基于检索和推理时修正，多集中在输入或输出层面，忽视了模型内部的信息精炼过程及早期层的作用。而对齐与微调类方法则资源消耗大。本文提出PLI（提前层插值），一种无需训练、可即插即用的干预方法，通过数学插值构建提前层，扩展大模型的信息处理深度与传递路径，受稳定扩散启发，增强事实一致性。在四个公开数据集上的实验表明，PLI能有效降低幻觉率，在多数情况下超越现有基线。进一步分析显示，层插值的成功与大模型内部机制密切相关。代码与数据集已开源：https://github.com/CuSO4-Chen/PLI。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate remarkable capabilities in text understanding and generation. However, their tendency to produce factually inconsistent outputs, commonly referred to as ''hallucinations'', remains a critical challenge. Existing approaches, such as retrieval-based and inference-time correction methods, primarily address this issue at the input or output level, often overlooking the intrinsic information refinement process and the role of premature layers. Meanwhile, alignment- and fine-tuning-based methods are resource-intensive. In this paper, we propose PLI (Premature Layers Interpolation), a novel, training-free, and plug-and-play intervention designed to enhance factuality. PLI mitigates hallucinations by inserting premature layers formed through mathematical interpolation with adjacent layers. Inspired by stable diffusion and sampling steps, PLI extends the depth of information processing and transmission in LLMs, improving factual coherence. Experiments on four publicly available datasets demonstrate that PLI effectively reduces hallucinations while outperforming existing baselines in most cases. Further analysis suggests that the success of layer interpolation is closely linked to LLMs' internal mechanisms. Our dataset and code are available at https://github.com/CuSO4-Chen/PLI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。