不依赖数据，直接在权重空间解耦MLP神经元，发现可解释的词汇通道。
Disentangling MLP Neuron Weights in Vocabulary Space
- 通过优化权重旋转以最大化词汇空间峰度，提取稀疏可读方向。
- 在Llama-3.1和Gemma-2模型上恢复的通道与神经元行为高度一致。
- 无需输入数据，适合大规模模型细粒度可解释性研究。
语言模型权重中的信息解读仍是机制可解释性的核心挑战。本文提出ROTATE（权重空间中旋转优化的词元对齐），一种无需前向传播的数据无关方法，可直接在权重空间解耦MLP神经元。关键观察是：编码连贯、单一语义概念的神经元在模型词汇空间上的投影具有高峰度。通过优化神经元权重的旋转以最大化其词汇空间峰度，该方法恢复出稀疏且可解释的方向，称为词汇通道。在Llama-3.1-8B-Instruct和Gemma-2-2B-it上的实验表明，ROTATE consistently recovered与神经元行为一致的词汇通道；移除单个通道会分别抑制对应输入激活或特定概念的生成。此外，通道级描述的聚合结果在头对头比较中优于优化的激活基线2-3倍。ROTATE提供了一种无数据的神经元权重分解，为语言模型的可解释性研究提供了可扩展、细粒度的构建模块。
原文摘要 · Abstract (English)
Interpreting the information encoded in language model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a data-free method requiring no forward passes that disentangles MLP neurons directly in weight space. Our approach relies on a key statistical observation: neurons that encode coherent, monosemantic concepts exhibit high kurtosis when projected onto the model's vocabulary. By optimizing rotations of neuron weights to maximize their vocabulary-space kurtosis, our method recovers sparse, interpretable directions which we name vocabulary channels. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it demonstrate that ROTATE consistently recovers vocabulary channels that are faithful to the neuron's behavior; ablating individual channels selectively disables corresponding input activations or the promotion of specific concepts. Moreover, aggregating channel-level descriptions yields comprehensive neuron descriptions that outperform optimized activation-based baselines by 2-3x in head-to-head comparisons. By providing a data-free decomposition of neuron weights, ROTATE offers a scalable, fine-grained building block for interpreting language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。