首次揭示大模型中记忆与推理由独立电路支持,可分别干预而不互相影响。
Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis
- 通过控制数据集和因果干预,定位记忆与推理的特定层、头和神经元
- 关闭记忆回路使事实召回准确率下降15%,推理能力不受影响;反之亦然
- 为安全可控的模型优化提供新思路,适合关注可解释性的研究者
基于Transformer的语言模型在记忆(检索已记住的事实)和推理(多步推断)方面表现优异,但二者是否依赖不同内部机制尚不明确。区分两者对预测模型泛化能力、设计针对性评估以及实现仅影响单一能力的安全干预至关重要。本文通过机制可解释性方法,利用合成语言谜题数据集,在层、注意力头和神经元层面探测Transformer模型。分析管道结合激活补丁和结构化消融,以因果方式衡量各组件对任务类型的影响。在两个模型家族(Qwen和LLaMA)中,我们发现对特定层和注意力头的干预会导致选择性损害:关闭识别出的“记忆回路”会使事实检索准确率最多下降15%,而推理能力保持不变;相反,关闭“推理回路”则使多步推理能力下降相当幅度。在神经元层面,观察到任务特异性放电模式,但效果较弱,与神经元的多义性一致。结果首次提供了因果证据,表明记忆与推理依赖于可分离但相互作用的电路。这些发现推动了机制可解释性发展,将电路结构与功能专门化关联起来,并展示了控制数据集和因果干预在揭示模型认知机制方面的潜力,有助于更安全地部署大语言模型。
原文摘要 · Abstract (English)
Transformer-based language models excel at both recall (retrieving memorized facts) and reasoning (performing multi-step inference), but whether these abilities rely on distinct internal mechanisms remains unclear. Distinguishing recall from reasoning is crucial for predicting model generalization, designing targeted evaluations, and building safer interventions that affect one ability without disrupting the other.We approach this question through mechanistic interpretability, using controlled datasets of synthetic linguistic puzzles to probe transformer models at the layer, head, and neuron level. Our pipeline combines activation patching and structured ablations to causally measure component contributions to each task type. Across two model families (Qwen and LLaMA), we find that interventions on distinct layers and attention heads lead to selective impairments: disabling identified "recall circuits" reduces fact-retrieval accuracy by up to 15\% while leaving reasoning intact, whereas disabling "reasoning circuits" reduces multi-step inference by a comparable margin. At the neuron level, we observe task-specific firing patterns, though these effects are less robust, consistent with neuronal polysemanticity.Our results provide the first causal evidence that recall and reasoning rely on separable but interacting circuits in transformer models. These findings advance mechanistic interpretability by linking circuit-level structure to functional specialization and demonstrate how controlled datasets and causal interventions can yield mechanistic insights into model cognition, informing safer deployment of large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。