用注意力分布优化大模型解码层选择，提升事实准确性
Attention-Guided Layer Selection for Contrastive Decoding in Large Language Models
- 利用自注意力机制结构信息指导解码层选择
- 在TruthfulQA上多答案指标提升，最优方案超越原DoLa
- 适合关注生成事实性、模型可解释性的研究者
对比解码方法如DoLa通过对比成熟层与早期层的输出分布，提升了大语言模型的事实性。然而，DoLa的动态层选择仅依赖输出词汇分布的差异。本文提出三种基于注意力的策略：Attention-JSD、Attention-Entropy-Max 和 Attention-Entropy-Min，利用内部自注意力机制携带的结构信息作为层选择信号。在TruthfulQA上的实验表明，尤其是Attention-JSD和Attention-Entropy-Min，持续优于原始DoLa。在多答案指标MC2和MC3上取得显著提升，说明注意力分布比输出词汇分布能更敏感地反映事实知识。
原文摘要 · Abstract (English)
Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa's dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies: Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min, which leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。