arXiv:2507.04404cs.AI2025-07被引 3

通过分析不同词元在模型层中的注意力模式,提升大模型生成事实准确性。

LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers

  • 根据词元类型动态调整各层注意力,实现精准控制
  • 在多个模型和基准上显著提升事实性,最高提升12.3%
  • 无需训练或修改模型,适合追求高可靠性的应用

大语言模型在自然语言理解和生成方面表现优异,但在知识密集型任务中仍易出现事实错误。现有解码阶段策略虽无需训练,但通常将词元级与层级信号分开处理,忽视了两者间的联合动态。本文提出一种词元感知、层内定位的对比解码方法,将特定词元类型与其最具影响力的Transformer层对齐,以改善事实生成。通过实证注意力分析发现:标点词元在早期层获得主导注意力,而概念词元则主导中间层语义推理。通过在对应深度有选择地抑制这些词元的注意力,诱导可控的事实退化,并据此生成对比信号以引导最终事实解码。该方法无需额外训练或模型修改,实验表明其在多个大模型和多种基准上均持续提升事实性。

原文摘要 · Abstract (English)

Large language models (LLMs) excel at natural language understanding and generation but remain vulnerable to factual errors, limiting their reliability in knowledge-intensive tasks. While decoding-time strategies provide a promising efficient solution without training, existing methods typically treat token-level and layer-level signals in isolation, overlooking the joint dynamics between them. In this work, we introduce a token-aware, layer-localized contrastive decoding method that aligns specific token types with their most influential transformer layers to improve factual generation. Through empirical attention analysis, we identify two key patterns: punctuation tokens receive dominant attention in early layers, while conceptual tokens govern semantic reasoning in intermediate layers. By selectively suppressing attention to these token types at their respective depths, we achieve the induction of controlled factual degradation and derive contrastive signals to guide the final factual decoding. Our method requires no additional training or model modification, and experiments demonstrate that our method consistently improves factuality across multiple LLMs and various benchmarks.

大模型事实性解码优化注意力分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。