arXiv:2604.00778cs.CL2026-04被引 2

模型算对了却答错,根源是后期信号被刻意压制。

From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks

  • 通过字符计数任务发现,正确答案在早期已编码但后期被抑制。
  • 关键错误来自最后几层MLP中的负向电路,会压低正确信号。
  • 适合关注模型可解释性与推理鲁棒性的研究者阅读。

大型语言模型(LLMs)在基础符号任务如单词字符计数上表现不佳,尽管在复杂基准上表现优异。我们以字符计数(如“apple中有几个p?”)为最小、可控的探测任务,隔离了词元级推理与高层干扰。在LLaMA、Qwen和Gemma等现代架构中,我们发现模型常在内部正确计算答案,却在输出层失败。通过探针分类器、激活修补、logit lens分析和注意力头追踪等机制分析,发现字符信息存在于早期和中期层表示中,但被后期少量组件(尤其是倒数第二和最终层的MLP)削弱。这些组件为负向电路:子网络会降低正确信号,倾向高概率但错误的输出。本研究提出两点贡献:第一,符号推理失败并非因缺乏表征或规模不足,而是模型计算图中的结构化干扰所致,解释了为何错误在扩展和指令微调下仍存甚至加剧;第二,模型前向传播呈现竞争解码机制,正确与错误假设共存并动态重加权,最终输出由抑制多于放大决定。这对可解释性与鲁棒性设计具有启示:简单符号任务暴露了现代大模型的深层弱点,亟需确保信息既编码又可靠使用的设计策略。

原文摘要 · Abstract (English)

Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks. Although this limitation has been noted, the internal reasons remain unclear. We use character counting (e.g., "How many p's are in apple?") as a minimal, controlled probe that isolates token-level reasoning from higher-level confounds. Using this setting, we uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma: models often compute the correct answer internally yet fail to express it at the output layer. Through mechanistic analysis combining probing classifiers, activation patching, logit lens analysis, and attention head tracing, we show that character-level information is encoded in early and mid-layer representations. However, this information is attenuated by a small set of components in later layers, especially the penultimate and final layer MLP. We identify these components as negative circuits: subnetworks that downweight correct signals in favor of higher-probability but incorrect outputs. Our results lead to two contributions. First, we show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph. This explains why such errors persist and can worsen under scaling and instruction tuning. Second, we provide evidence that LLM forward passes implement a form of competitive decoding, in which correct and incorrect hypotheses coexist and are dynamically reweighted, with final outputs determined by suppression as much as by amplification. These findings carry implications for interpretability and robustness: simple symbolic reasoning exposes weaknesses in modern LLMs, underscoring need for design strategies that ensure information is encoded and reliably used.

模型可解释性符号推理负向电路机制分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。