揭示大模型算术能力的神经机制，找到关键注意力头与可解释神经元。
Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis
- 通过对比神经元分析定位算术运算专用注意力头。
- 发现浅层前馈网络增强特征、深层网络提升预测概率。
- 可解释模型微调机制，适合模型可解释性研究者。
我们发现算术能力集中在少数注意力头中，每个头专精于不同运算。为深入探究原因，提出对比神经元分析（CNA）方法，识别出从输入到预测的四个阶段：浅层前馈网络（FFN）增强特征、浅层注意力层传递特征、算术注意力头预测结果、深层FFN增强预测。此外，识别出特征增强与特征预测阶段中人类可解释的FFN神经元。基于此，研究了LoRA机制，发现其通过放大与预测相关的FFN神经元系数得分来提升预测概率。最后，将该方法应用于算术任务的模型剪枝及性别偏见削减的模型编辑。代码已开源。
原文摘要 · Abstract (English)
We find arithmetic ability resides within a limited number of attention heads, with each head specializing in distinct operations. To delve into the reason, we introduce the Comparative Neuron Analysis (CNA) method, which identifies an internal logic chain consisting of four distinct stages from input to prediction: feature enhancing with shallow FFN neurons, feature transferring by shallow attention layers, feature predicting by arithmetic heads, and prediction enhancing among deep FFN neurons. Moreover, we identify the human-interpretable FFN neurons within both feature-enhancing and feature-predicting stages. These findings lead us to investigate the mechanism of LoRA, revealing that it enhances prediction probabilities by amplifying the coefficient scores of FFN neurons related to predictions. Finally, we apply our method in model pruning for arithmetic tasks and model editing for reducing gender bias. Code is on https://github.com/zepingyu0512/arithmetic-mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。