通过输入输出关系揭示Transformer中门控神经元的运作机制。
Understanding Gated Neurons in Transformers from Their Input-Output Functionality
- 分析神经元输入与输出权重的余弦相似度，识别其增益或削弱特征方向。
- 早期到中期层以增强型神经元为主，后期层更倾向抑制型神经元。
- 揭示概念表征丰富化是事实回忆的初始步骤，适合模型可解释性研究者。
可解释性研究通常基于神经元激活上下文及其输出权重向量来理解语言模型中的MLP神经元，但对输入与输出之间相互作用的关注较少。例如，当神经元检测到输入中的某一方向时，可能向残差流添加相同方向（增强型神经元），或减弱其存在（抑制型神经元）。本文通过考察神经元输入与输出权重间的余弦相似度，系统分析了这一交互机制。我们在12个不同模型上应用该方法，发现早期至中期层以增强型神经元占主导，而后期层则更多表现为抑制型。我们进一步提出，增强型神经元主要负责丰富概念表征，这是事实回忆的第一步。该输入-输出视角补充了依赖激活行为的分析方法，也超越了将输入与输出分离处理的传统思路。
原文摘要 · Abstract (English)
Interpretability researchers have attempted to understand MLP neurons of language models based on both the contexts in which they activate and their output weight vectors. They have paid little attention to a complementary aspect: the interactions between input and output. For example, when neurons detect a direction in the input, they might add much the same direction to the residual stream ("enrichment neurons") or reduce its presence ("depletion neurons"). We address this aspect by examining the cosine similarity between input and output weights of a neuron. We apply our method to 12 models and find that enrichment neurons dominate in early-middle layers whereas later layers tend more towards depletion. To explain this finding, we argue that enrichment neurons are largely responsible for enriching concept representations, one of the first steps of factual recall. Our input-output perspective is a complement to activation-dependent analyses and to approaches that treat input and output separately.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。