提出新机制提升大模型多跳推理能力，让单层模型达到双层效果。
Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models
- 通过追踪逻辑值传播路径，揭示推理的四个阶段。
- 发现多跳推理失败主因是关系属性提取阶段冲突逻辑。
- 设计反向注意力机制，显著提升多个大模型的推理准确率。
我们研究大语言模型在类似“莫扎特母亲的配偶是谁”这类提示中如何进行隐式多跳推理。为分析该过程，引入日志流（logit flow）这一可解释性方法，追踪逻辑值在各层与位置间的传播路径。利用日志流，识别出单跳知识预测的四个阶段：(A) 实体主体增强，(B) 实体属性提取，(C) 关系主体增强，(D) 关系属性提取。将分析扩展至多跳推理后发现，失败常源于关系属性提取阶段，因冲突逻辑导致准确率下降。为此，提出反向注意力（back attention），使低层能利用高层不同位置的隐藏状态参与注意力计算。实验显示，加入反向注意力后，单层Transformer性能达到双层水平。在四个主流大模型上应用，其在五个推理数据集上均取得性能提升，验证了该机制对增强隐式多跳推理的有效性。
原文摘要 · Abstract (English)
We investigate how large language models perform latent multi-hop reasoning in prompts like "Wolfgang Amadeus Mozart's mother's spouse is". To analyze this process, we introduce logit flow, an interpretability method that traces how logits propagate across layers and positions toward the final prediction. Using logit flow, we identify four distinct stages in single-hop knowledge prediction: (A) entity subject enrichment, (B) entity attribute extraction, (C) relation subject enrichment, and (D) relation attribute extraction. Extending this analysis to multi-hop reasoning, we find that failures often stem from the relation attribute extraction stage, where conflicting logits reduce prediction accuracy. To address this, we propose back attention, a novel mechanism that enables lower layers to leverage higher-layer hidden states from different positions during attention computation. With back attention, a 1-layer transformer achieves the performance of a 2-layer transformer. Applied to four LLMs, back attention improves accuracy on five reasoning datasets, demonstrating its effectiveness in enhancing latent multi-hop reasoning ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。