Transformer模型的数值表征噪声随数值增大而减小，与生物系统相反。
Same Geometry, Opposite Noise: Transformer Magnitude Representations Lack Scalar Variability
- 分析26个数值在3个大模型中的隐藏状态，发现噪声随数值增大而降低
- 噪声与数值大小呈负相关（斜率约-0.19），且在不同维度中显著不同
- 结果表明纯分布学习无法生成生物系统中的恒定变异系数特征
标量可变性——即表征噪声与数值大小成比例，保持恒定变异系数——是生物数值系统的重要特征。我们通过分析26个数值在三个7-8B参数模型（Llama-3-8B-Instruct、Mistral-7B-Instruct-v0.3、Llama-3-8B-Base）中载体句子的隐藏状态分散情况，检验了Transformer语言模型是否具备该特性（数据来自Cacioli, 2026）。结果发现相反现象：沿数值轴，表征可变性随数值增大而降低（斜率α≈-0.19；三模型中仅0/16主层α>0）。该负相关在全维空间（α≈-0.04）和去除句子身份影响后（α≈-0.007）仍一致。沿数值轴的反标量模式比正交维度强3-5倍，语料频率能强力预测各数值的可变性（rho=.84）。结果表明，仅靠分布学习不足以产生标量可变性：Transformer复制了对数压缩的数值几何结构，但未再现生物系统中的恒定-变异系数噪声特征。
原文摘要 · Abstract (English)
Scalar variability -- the finding that representational noise scales proportionally with magnitude, producing a constant coefficient of variation -- is a hallmark of biological magnitude systems. We tested whether transformer language models exhibit this property by analysing the dispersion of hidden-state representations across carrier sentences for 26 numerical magnitudes in three 7-8B parameter models (Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3-8B-Base; data from Cacioli, 2026). We found the opposite: representational variability decreased with magnitude along the magnitude axis (scaling exponent alpha approx -0.19; 0/16 primary layers with alpha > 0, all three models). The negative sign was consistent in full-dimensional space (alpha approx -0.04) and after sentence-identity correction (alpha approx -0.007). The anti-scalar pattern was 3-5x stronger along the magnitude axis than orthogonal dimensions, and corpus frequency strongly predicted per-magnitude variability (rho = .84). These results demonstrate that distributional learning alone is insufficient to produce scalar variability: transformers reproduce log-compressive magnitude geometry but not the constant-CV noise signature observed in biological systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。