发现大模型内部数字表示呈对数分布，类似人类认知机制。
Number Representations in LLMs: A Computational Parallel to Human Perception
- 通过主成分分析与几何回归，探测模型各层数字编码结构。
- 数字间距随数值增大而缩小，符合对数规律，非线性压缩表征。
- 为理解模型认知机制提供新视角，适合关注神经表征的研究者。
人类被认为在对数心理数轴上感知数字，小数值的分辨率高于大数值。这一认知偏见由神经科学和行为研究支持，表明数值大小以非线性方式处理，而非均匀线性尺度。受此启发，我们探究大语言模型（LLMs）内部数字表示是否也存在类似对数结构。通过分析模型不同层中数值的编码方式，采用主成分分析（PCA）与偏最小二乘法（PLS）结合几何回归，揭示嵌入空间中的潜在结构。结果表明，模型的数字表示呈现亚线性间距，数值间距离与对数尺度一致。这暗示大语言模型可能像人类一样，以压缩、非均匀的方式编码数字。
原文摘要 · Abstract (English)
Humans are believed to perceive numbers on a logarithmic mental number line, where smaller values are represented with greater resolution than larger ones. This cognitive bias, supported by neuroscience and behavioral studies, suggests that numerical magnitudes are processed in a sublinear fashion rather than on a uniform linear scale. Inspired by this hypothesis, we investigate whether large language models (LLMs) exhibit a similar logarithmic-like structure in their internal numerical representations. By analyzing how numerical values are encoded across different layers of LLMs, we apply dimensionality reduction techniques such as PCA and PLS followed by geometric regression to uncover latent structures in the learned embeddings. Our findings reveal that the model's numerical representations exhibit sublinear spacing, with distances between values aligning with a logarithmic scale. This suggests that LLMs, much like humans, may encode numbers in a compressed, non-uniform manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。