分析Transformer模型中GLU神经元的激活模式，揭示四种符号组合的功能差异。
GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Models
- 基于SwiGLU等门控激活函数，区分门控与输入激活的正负组合。
- 可视化每种符号组合对应的文本实例及其出现频率。
- 适合关注模型可解释性的研究者使用，助力发现新规律。
我们提出GLUScope，一个开源工具，用于分析基于Transformer的语言模型中的神经元，专为可解释性研究设计。与以往工具不同，本工具聚焦更近期的模型，特别是采用如SwiGLU等门控激活函数的模型。这带来新挑战：仅分析正激活已不够。因为神经元的门控与输入激活均可正可负，形成四种不同的符号组合，某些情况下功能迥异。因此，对任一神经元，该工具展示每种符号组合对应的文本示例，并标注其出现频率。我们描述了该工具如何催生新见解。演示地址：https://sjgerstner.github.io/gluscope。
原文摘要 · Abstract (English)
We present GLUScope, an open-source tool for analyzing neurons in Transformer-based language models, intended for interpretability researchers. We focus on more recent models than previous tools do; specifically we consider gated activation functions such as SwiGLU. This introduces a new challenge: understanding positive activations is not enough. Instead, both the gate and the in activation of a neuron can be positive or negative, leading to four different possible sign combinations that in some cases have quite different functionalities. Accordingly, for any neuron, our tool shows text examples for each of the four sign combinations, and indicates how often each combination occurs. We describe examples of how our tool can lead to novel insights. A demo is available at https: //sjgerstner.github.io/gluscope.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。