大模型处理数字时,在位数边界出现类范畴感知的几何畸变。
Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries
- 在阿拉伯数字处理中,模型隐藏层出现类别边界几何畸变。
- 100%模型层在10、100等位数边界处表现出显著区分增强。
- 无需语义类别知识,仅输入结构断点即可引发此类现象。
范畴感知(CP)——在类别边界处增强辨别能力——是知觉心理学中研究最深入的现象之一。本文发现,大型语言模型(LLMs)在处理阿拉伯数字时,其隐藏状态表征中也出现类似的几何畸变。通过跨六种模型、五种架构家族的表示相似性分析,研究显示:在所有测试模型的主层中,一个包含对数距离加边界增强的CP加性模型,比纯连续模型更准确地拟合表征几何。该效应特异性体现在结构定义的边界(如10和100的位数转换),在非边界对照位置及无分词断点的语言类别(如冷热)温度域中均未出现。两种不同特征浮现:‘经典CP’(Gemma、Qwen)表现为显式分类与几何畸变并存;‘结构CP’(Llama、Mistral、Phi)则仅体现几何畸变,模型无法报告类别差异。这种分离在边界间稳定存在,属于架构特性而非刺激依赖。输入格式的结构性断点足以在大模型中诱发范畴感知几何,无需明确的语义类别知识。
原文摘要 · Abstract (English)
Categorical perception (CP) -- enhanced discriminability at category boundaries -- is among the most studied phenomena in perceptual psychology. This paper reports that analogous geometric warping occurs in the hidden-state representations of large language models (LLMs) processing Arabic numerals. Using representational similarity analysis across six models from five architecture families, the study finds that a CP-additive model (log-distance plus a boundary boost) fits the representational geometry better than a purely continuous model at 100% of primary layers in every model tested. The effect is specific to structurally defined boundaries (digit-count transitions at 10 and 100), absent at non-boundary control positions, and absent in the temperature domain where linguistic categories (hot/cold) lack a tokenisation discontinuity. Two qualitatively distinct signatures emerge: "classic CP" (Gemma, Qwen), where models both categorise explicitly and show geometric warping, and "structural CP" (Llama, Mistral, Phi), where geometry warps at the boundary but models cannot report the category distinction. This dissociation is stable across boundaries and is a property of the architecture, not the stimulus. Structural input-format discontinuities are sufficient to produce categorical perception geometry in LLMs, independently of explicit semantic category knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。