发现大模型多跳推理中层序反转现象，提出概率召回-提取机制解释。
Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models
- 用概率召回与选择性提取框架解释多跳推理机制。
- 发现后期跳跃答案比桥接实体更早可解，且跳数越多越明显。
- 适合研究模型内部推理过程或失败原因的读者参考。
大型语言模型在多跳推理任务中表现良好，但其内部如何组合多个事实仍不明确。近期研究提出‘跳对齐电路假说’，认为桥接实体在各层中按顺序计算，之后才生成后续跳跃的答案。通过对真实世界多跳查询的系统分析，我们发现该假设不具备普遍性：后期跳跃的答案实体可能比桥接实体更早变得可解，这一现象称为‘层序反转’，且随着总跳数增加而增强。为解释此行为，我们提出‘概率回忆-提取’框架，将多跳推理建模为浅层MLP层中的广泛概率回忆，随后在深层注意力层进行选择性提取。该框架通过系统的探针分析得到实证验证，重新解释了先前的层间解码证据，说明了思维链带来的提升，并提供了尽管单跳知识正确但仍出现多跳失败的机制诊断。代码已公开于 https://github.com/laquabe/Layer-Order-Inversion。
原文摘要 · Abstract (English)
Large language models (LLMs) perform well on multi-hop reasoning, yet how they internally compose multiple facts remains unclear. Recent work proposes \emph{hop-aligned circuit hypothesis}, suggesting that bridge entities are computed sequentially across layers before later-hop answers. Through systematic analyses on real-world multi-hop queries, we show that this hop-aligned assumption does not generalize: later-hop answer entities can become decodable earlier than bridge entities, a phenomenon we call \emph{layer-order inversion}, which strengthens with total hops. To explain this behavior, we propose a \emph{probabilistic recall-and-extract} framework that models multi-hop reasoning as broad probabilistic recall in shallow MLP layers followed by selective extraction in deeper attention layers. This framework is empirically validated through systematic probing analyses, reinterpreting prior layer-wise decoding evidence, explaining chain-of-thought gains, and providing a mechanistic diagnosis of multi-hop failures despite correct single-hop knowledge. Code is available at https://github.com/laquabe/Layer-Order-Inversion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。