arXiv:2603.05143cs.CLcs.LG2026-03

揭示Transformer模型中类比推理的内在机制:共享特征促成属性迁移。

Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers

  • 通过联合训练相似性与属性数据,使模型实现类比推理。
  • 顺序训练需先学相似结构,再学具体属性,存在课程不对称性。
  • 两跳推理本质是带恒等桥的类比,训练数据显式包含此类模式。

大语言模型的推理理解因多种推理类型混杂而复杂。本文聚焦类比推理——即模型将一个实体的属性迁移到具有已知共性另一实体上,并研究该能力如何在训练中产生。为使问题可分析,我们构建了一个极简的Transformer抽象模型,仅关注学习表征如何支持类比推理。理论证明三个关键结果:第一,联合训练相似性和属性前提可使表示对齐,从而实现类比推理;第二,顺序训练仅当先学习相似结构、后学具体属性时成功,揭示课程不对称性;第三,在简化设定中,两跳推理(a→b, b→c ⇒ a→c)可视为以身份桥(b=b)为中介的类比推理,且此类桥接在训练数据中显式出现。三者共同揭示统一机制:共享属性使实体在表征空间中对齐,从而通过特征相似实现属性迁移。80亿参数模型实验表明理论与现象定性一致,提示表征几何在类比推理中起关键作用,超越简化模型范围。

原文摘要 · Abstract (English)

Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoning, where a model transfers an attribute between entities that share known properties, and study when such transfer can emerge from training. To make the problem analytically tractable, we study a minimal transformer-style abstraction that isolates how learned representations support analogical reasoning. Within this setting, we prove three key results. First, joint training on similarity and attribution premises enables analogical reasoning through aligned representations. Second, sequential training succeeds only when similarity structure is learned before specific attributes, revealing a curriculum asymmetry. Third, in our stylized setting, two-hop reasoning $(a \to b, b \to c \Rightarrow a \to c)$ can be viewed as analogical reasoning with identity bridges $(b=b)$, which appear explicitly in training data. Together, these results reveal a unified mechanism: entities with shared properties become aligned in representation space, enabling property transfer through feature resemblance. Experiments with architectures up to 8B parameters show qualitative agreement with the theory and suggest that representational geometry plays an important role in analogical reasoning beyond the stylized model.

类比推理Transformer表征对齐认知机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。