Transformer通过相邻比较学会数轴式排列,自然产生人类认知中的距离效应。
Emergent Ordinal Geometry in Transformers Trained on Local Comparisons

- 用相邻比较训练模型,让嵌入向量自发形成一维数轴结构。
- 远距离比较判断更准确,且置信度随距离递增,符合心理实验规律。
- 大模型中也发现类似几何结构,可反映不同类型的序关系。
传递推理是通过已知相邻关系(A < B, B < C)推断出非相邻关系(A < C)的挑战。人类和动物并非通过逻辑链推理,而是依赖类比心理数轴,其特征是符号距离效应:相距越远的比较越容易。本文研究小规模Transformer在仅学习隐藏全序下的相邻比较后,对未见远距离对的泛化能力。结果发现,模型在分布外泛化能力出现的同时,实体嵌入发生显著几何重组:嵌入坍缩至一维流形,主轴几乎完美恢复隐藏排序,并表现出类似grokking的瞬态优化动态。关键的是,即使准确率已达上限,决策置信度与几何分离度仍随排名距离单调增加,直接复现了跨人类、灵长类和啮齿类多年行为实验中的符号距离效应。进一步发现在预训练大语言模型中,同样存在对齐排序的几何结构,能捕捉不同序关系拓扑:大小与数字为线性,月份为循环。这些结果将50年来的行为规律建立在学习表示的几何基础上,为传递推理提供了连接认知科学与现代神经网络的机制解释。
原文摘要 · Abstract (English)
Transitive inference is the challenge of inferring that A < C from knowing only adjacent relations (A < B, B < C). It is solved by humans and animals not through logical chaining but via an analogue mental number line, whose signature is the symbolic distance effect: distant comparisons are easier than nearby ones. We ask whether Transformers acquire the same primitive, training small models exclusively on adjacent comparisons from a hidden total order and evaluating generalization to unseen distant pairs. We find that out-of-distribution generalization emerges alongside a striking geometric reorganization: entity embeddings collapse onto a one-dimensional manifold whose principal axis recovers the hidden rank order with near-perfect fidelity, and this structure is sensitive to optimization in ways that produce grokking-like transient dynamics. Critically, even when accuracy is at ceiling, decision confidence and geometric separation both scale monotonically with rank distance, directly mirroring the symbolic distance effect observed across decades of behavioural experiments on humans, primates, and rodents. We further show the same rank-aligned geometry in a pretrained large language model, where it tracks the topology of each ordinal relation: linear for sizes and digits, cyclic for months. These results ground a 50-year-old behavioural regularity in the geometry of learned representations, offering a mechanistic account of transitive inference that bridges cognitive science and modern neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。