通过分析注意力分布，从GPT-2小模型中挖掘出对上下文敏感的神经元。
Decomposing Attention To Find Context-Sensitive Neurons
- 利用校准文本采样软最大值分母，稳定多头注意力输出。
- 仅凭权重和单一校准文本，就可近似第一层注意力组合输出。
- 发现数百个响应高阶语境的神经元，包括未在校准文本中激活的。
我们研究Transformer语言模型，分析注意力分布分散且注意力分数对内容依赖弱的注意力头。我们认为，在固定词元分布时，这些头的软最大值分母是稳定的。通过从‘校准文本’中采样软最大值分母，我们可以将GPT2-Small第一层中多个此类稳定头的输出组合起来，用周围文本的线性摘要近似其联合输出。该近似使我们仅凭权重和一个校准文本，即可揭示数百个对周围文本高层上下文属性有响应的第一层神经元，包括在该校准文本中未激活的神经元。
原文摘要 · Abstract (English)
We study transformer language models, analyzing attention heads whose attention patterns are spread out, and whose attention scores depend weakly on content. We argue that the softmax denominators of these heads are stable when the underlying token distribution is fixed. By sampling softmax denominators from a "calibration text", we can combine together the outputs of multiple such stable heads in the first layer of GPT2-Small, approximating their combined output by a linear summary of the surrounding text. This approximation enables a procedure where from the weights alone - and a single calibration text - we can uncover hundreds of first layer neurons that respond to high-level contextual properties of the surrounding text, including neurons that didn't activate on the calibration text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。