通过身份桥机制，让模型学会跨跳推理的通用规律。
Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge
- 引入身份桥监督，强制桥接词保持原样映射。
- 仅用单层Transformer就能实现跨域两跳推理泛化。
- 适合研究模型推理机制或提升泛化能力的读者。
大型语言模型在分布内多跳推理表现优异，但在未见组合上失败，这一现象称为两跳推理困境。本文认为其根源在于缺乏对桥接实体的监督。为此提出身份桥（identity bridge），一种最小化监督方式，强制桥接标记保持身份映射。在此监督下，即使是一层采用均匀注意力的Emb-MLP模型，也能实现分布外（OOD）的两跳泛化。理论分析表明，身份桥产生隐式正则化，促使模型建立从主体到答案的直接关联。实证结果显示，标准GPT-2模型的表现与简单Emb-MLP模型在不同问题复杂度下高度一致。对主流LLM微调后的分析进一步表明，正确的两跳预测始终伴随主体到答案关系的建立，验证了结论在真实场景中的适用性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel at multi-hop reasoning in distribution, yet fail on unseen compositions, a phenomenon known as the curse of two-hop reasoning. In this work, we argue that this phenomenon can be attributed to a missing supervision on the bridge entity. We formalize this gap by introducing identity bridge, a minimal supervision that enforces a identity mapping on bridge tokens. Under this supervision, even a one-layer transformer with uniform attention (Emb-MLP) can achieve out-of-distribution (OOD) two-hop generalization. We provide a theoretical analysis demonstrating that identity bridge induces an implicit regularization effect, leading the model to establish a direct subject-to-answer association. From an empirical perspective, the performance of standard GPT-2 models aligns closely with simple Emb--MLP models across varying levels of problem complexity. Finally, analyses of fine-tuned mainstream LLMs indicate that correct two-hop predictions consistently coincide with the establishment of a subject-to-answer relationship, extending our findings to realistic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。