语言模型通过几何结构实现文本行断行计数，揭示其隐含视觉感知机制。
When Models Manipulate Manifolds: The Geometry of a Counting Task
- 用低维曲面表示字符数，类比生物位置细胞的离散特征家族
- 注意力头通过几何变换估算到行边界的距离，实现精确断行预测
- 发现可被劫持的视觉幻觉，适合关注模型可解释性与几何视角的研究者
语言模型虽仅接收词元序列，却能感知文本的视觉属性。我们系统研究了 Claude 3.5 Haiku 如何完成固定宽度文本的换行任务。结果发现，字符数量在低维弯曲流形上以稀疏特征族离散表示，类似生物中的位置细胞。准确的断行预测依赖一系列几何变换：将词元长度累积至字符计数流形，注意力头对这些流形进行扭转以估计距行边界距离，最终通过将估计值正交排列形成线性决策边界来触发断行。我们通过因果干预验证了这一机制，并发现了可劫持计数机制的视觉幻觉。本工作揭示了早期层丰富的感官处理能力、注意力算法的复杂性，以及融合特征与几何视角在可解释性研究中的重要性。
原文摘要 · Abstract (English)
Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。