通过剪枝构建对比模型，提升大模型生成事实性。
PruneCD: Contrasting Pruned Self Model to Improve Decoding Factuality
- 用层剪枝构建弱化版模型作为对比基准
- 在多个数据集上显著提升事实准确率，推理开销极低
- 适合需要高可靠性文本生成的场景
为缓解大语言模型的幻觉问题，现有方法DoLa利用同一模型的早期退出输出作为对比先验。但我们发现这些早期输出往往平坦、幅度小，难以体现有效对比。为此，我们提出PruneCD，一种新型对比解码方法：通过层剪枝构建弱化模型，而非依赖早期退出。该设计使输出概率更具信息量且与主模型对齐更佳，从而实现更有效的对比解码。定性与定量分析表明，PruneCD在保持极低推理开销的前提下，持续提升生成内容的事实性，提供了一种鲁棒且实用的幻觉抑制方案。
原文摘要 · Abstract (English)
To mitigate the hallucination problem in large language models, DoLa exploits early exit logits from the same model as a contrastive prior. However, we found that these early exit logits tend to be flat, low in magnitude, and fail to reflect meaningful contrasts. To address this, we propose PruneCD, a novel contrastive decoding method that constructs the amateur model via layer pruning rather than early exit. This design leads to more informative and well-aligned logits, enabling more effective contrastive decoding. Through qualitative and quantitative analyses, we demonstrate that PruneCD consistently improves factuality with minimal inference overhead, offering a robust and practical approach to mitigating hallucinations in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。