发现语言模型的MLP神经元具有稀疏性，可直接用于追踪因果神经元电路。
Language Model Circuits Are Sparse in the Neuron Basis
- 用梯度方法在MLP神经元基上实现端到端电路追踪
- 仅需约100个神经元即可控制主谓一致任务表现
- 能定位并操控特定推理步骤，适合模型可解释性研究
神经网络中的高层概念未必对应单个神经元（Smolensky, 1986）。为提升语言模型可解释性，现有研究转向将神经元基分解为更可解释的计算单元，如稀疏自编码器（SAEs）。然而，并非所有神经元表示都不可解释。本文首次实证表明，MLP神经元作为特征基与SAEs一样稀疏。基于此，我们构建了一个端到端的基于梯度的归因管道，在MLP神经元基上实现电路追踪，成功识别出多种任务下的因果有效神经元。在标准主谓一致基准测试（Marks et al., 2025）中，约10²个MLP神经元组成的电路即可控制模型行为；在多跳城市-国家-首都任务（Lindsey et al., 2025）中，发现小规模神经元集合可编码特定隐式推理步骤（如城市→所属州），且可通过调控改变模型输出。该工作在不增加训练成本的前提下，推进了语言模型的自动化可解释性研究。
原文摘要 · Abstract (English)
The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques which decompose the neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that MLP neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for circuit tracing on the MLP neuron basis, which surfaces causally effective neurons on a variety of tasks. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx 10^2$ MLP neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of language models without imposing additional training costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。