通过模型输出层直接解析大模型神经元，10秒内定位特定概念神经元。
Flash Interpretability: Decoding Specialised Feature Neurons in Large Language Models with the LM-Head
- 利用模型末尾的LM-head将神经元权重映射为词概率，实现直接解码。
- 在Llama 3.1 8B中发现超过75%的指令模型神经元与预训练模型共享相同顶级词。
- 可快速定位如'狗'或'加州'等概念神经元，并通过抑制控制输出内容。
大语言模型通常拥有数十亿参数，其运行机制难以解释。本文展示可通过模型最终投影层（LM-head）直接将神经元权重解码为词概率。以Llama 3.1 8B为例，我们利用该方法识别出如'狗'神经元、'加州'神经元等专用特征神经元，并通过固定这些神经元验证其对输出概率的影响。在预训练与指令微调模型上评估，发现指令模型中超过75%的上投影层神经元与其预训练版本具有相同的最高相关词。进一步实验表明，固定'狗'神经元后，指令模型在被问及最喜欢动物时始终生成关于狗的内容。本方法可在不到10秒内完成对整个Llama 3.1 8B模型上投影层神经元的特征映射，计算开销极低。
原文摘要 · Abstract (English)
Large Language Models (LLMs) typically have billions of parameters and are thus often difficult to interpret in their operation. In this work, we demonstrate that it is possible to decode neuron weights directly into token probabilities through the final projection layer of the model (the LM-head). This is illustrated in Llama 3.1 8B where we use the LM-head to find examples of specialised feature neurons such as a "dog" neuron and a "California" neuron, and we validate this by clamping these neurons to affect the probability of the concept in the output. We evaluate this method on both the pre-trained and Instruct models, finding that over 75% of neurons in the up-projection layers in the instruct model have the same top associated token compared to the pretrained model. Finally, we demonstrate that clamping the "dog" neuron leads the instruct model to always discuss dogs when asked about its favourite animal. Through our method, it is possible to map the top features of the entirety of Llama 3.1 8B's up-projection neurons in less than 10 seconds, with minimal compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。