揭示Transformer模型如何通过几何对齐和函数映射实现类比推理
Emergent Analogical Reasoning in Transformers
- 用范畴论中的函子概念形式化类比推理，构建可控制的评测任务
- 发现模型规模、数据特征和优化方式显著影响类比能力的涌现
- 证明类比推理由嵌入空间的结构对齐与函数变换共同实现
类比是人类智能的核心能力，使一个领域中发现的抽象模式能迁移到另一领域。尽管其在认知中至关重要，但Transformer如何习得并执行类比推理仍不清楚。受范畴论中函子概念启发，我们形式化类比推理为跨范畴实体间对应关系的推断。基于此，我们设计了可控环境下评估类比推理涌现的合成任务。结果表明，类比推理的涌现高度依赖于数据特性、优化选择和模型规模。通过机制分析，我们发现Transformer中的类比推理可分解为两个关键组件：(1) 嵌入空间中关系结构的几何对齐；(2) Transformer内部的函子应用。这些机制使模型能够将关系结构从一个范畴转移到另一个范畴，从而实现类比。最后，我们量化了这些效应，并发现相同趋势也存在于预训练的大语言模型中。本研究将类比从抽象认知概念转变为现代神经网络中可机制化解释的现象。
原文摘要 · Abstract (English)
Analogy is a central faculty of human intelligence, enabling abstract patterns discovered in one domain to be applied to another. Despite its central role in cognition, the mechanisms by which Transformers acquire and implement analogical reasoning remain poorly understood. In this work, inspired by the notion of functors in category theory, we formalize analogical reasoning as the inference of correspondences between entities across categories. Based on this formulation, we introduce synthetic tasks that evaluate the emergence of analogical reasoning under controlled settings. We find that the emergence of analogical reasoning is highly sensitive to data characteristics, optimization choices, and model scale. Through mechanistic analysis, we show that analogical reasoning in Transformers decomposes into two key components: (1) geometric alignment of relational structure in the embedding space, and (2) the application of a functor within the Transformer. These mechanisms enable models to transfer relational structure from one category to another, realizing analogy. Finally, we quantify these effects and find that the same trends are observed in pretrained LLMs. In doing so, we move analogy from an abstract cognitive notion to a concrete, mechanistically grounded phenomenon in modern neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。