语言模型中数字概念的表征具有稳定关系结构,支持通用性与任务灵活性。
The Representational Geometry of Number
- 通过几何关系而非概念本身实现表征共享
- 不同任务的数字表征位于可线性转换的子空间中
- 适合研究认知科学与大模型表征机制的读者
认知科学的核心问题之一是概念表征是否收敛到共享流形以支持泛化,还是分化到正交子空间以减少任务干扰。尽管先前研究发现两者均有证据,但缺乏对这些特性如何共存并随任务变化的机制解释。本文以数字概念为测试案例,利用语言模型作为高维计算基底,发现数字表征在不同任务间保持稳定的相对关系结构。特定任务的表征嵌入于不同的子空间,低层特征如数值大小和奇偶性沿可分离的线性方向编码。关键发现是,这些子空间可通过线性映射相互转换,表明尽管表征位于不同子空间,其关系结构仍高度共享。结果揭示了语言模型在保持数字表征共享结构的同时实现功能灵活性的机制,提示理解源于对共享关系结构施加任务特异性变换。
原文摘要 · Abstract (English)
A central question in cognitive science is whether conceptual representations converge onto a shared manifold to support generalization, or diverge into orthogonal subspaces to minimize task interference. While prior work has discovered evidence for both, a mechanistic account of how these properties coexist and transform across tasks remains elusive. We propose that representational sharing lies not in the concepts themselves, but in the geometric relations between them. Using number concepts as a testbed and language models as high-dimensional computational substrates, we show that number representations preserve a stable relational structure across tasks. Task-specific representations are embedded in distinct subspaces, with low-level features like magnitude and parity encoded along separable linear directions. Crucially, we find that these subspaces are largely transformable into one another via linear mappings, indicating that representations share relational structure despite being located in distinct subspaces. Together, these results provide a mechanistic lens of how language models balance the shared structure of number representation with functional flexibility. It suggests that understanding arises when task-specific transformations are applied to a shared underlying relational structure of conceptual representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。