大模型计数失败不是因为记不住，而是某层神经元故意删掉正确答案。
Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
- 在残差流中能精准解码重复词数，说明模型内部有正确表征
- 约85%-93%深度的MLP层会把正确计数替换成错误结果
- 该问题仅出现在空格分隔的词重复场景，适合研究模型推理机制
大型语言模型在统计列表中单词重复次数时表现不佳，尽管其在更复杂的推理任务中表现良好。这一现象常被归因于内部计数追踪能力的局限。我们发现该归因错误：残差流中的线性探针在每个嵌入后层都能近乎完美地解码出正确计数，且在输出出现错误的层中依然保持正确。注意力模式未显示重复词上的信息坍缩，分词误差也并非原因。相反，在网络深度约85%-93%处的一个多层感知机（MLP）模块会将正确计数覆盖为固定错误答案。移除该模块可改变错误输出，证明其因果责任。该模块仅对空格分隔的重复词触发，对重复数字不生效。该模式在Llama-3.2（1B和3B）与Qwen2.5（1.5B、3B和7B）中均在一致相对深度下成立。计数信息被正确表示，但特定计算阻止其到达输出，因此表征失败与路由失败需不同干预策略。
原文摘要 · Abstract (English)
Large language models fail at counting how many times a word repeats in a list, even though they perform well on far harder reasoning tasks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer and they do so even at the exact layers where the wrong answer crystallizes in the output. Attention patterns show no evidence of collapse over repeated tokens and tokenization artifacts account for none of the failure. Instead, a multi-layer perceptron (MLP) block at roughly 85--93\% network depth overwrites the correctly-encoded count with a fixed wrong answer. Ablating this block changes the wrong output and establishes it as causally responsible for the failure. The block fires on the space-separated repeated-word format and is absent for repeated digit-tokens. The pattern holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. The count is represented correctly and a specific computation prevents it from reaching the output, so representation failures and routing failures require different interventions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。