提出新方法定位大模型中负责特定概念的神经元,显著降低计算量。
NEAT: Concept driven Neuron Attribution in LLMs
- 用概念向量识别与特定概念相关的神经元,实现精准定位。
- 将计算量从O(n×m)降至O(n),效率大幅提升。
- 可应用于检测和关闭偏见神经元,适合模型可解释性研究者。
定位对最终预测起关键作用的神经元,是理解大语言模型内部机制、打开黑箱的重要一步。以往方法虽尝试在神经元层面探查机制,但难以有效表征概念,且计算开销大。本文借助概念向量,提出一种定位关键概念神经元的新方法:若神经元数量为n,样本数为m,相比之前方法的O(n×m)前向传播次数,仅需O(n)次,大幅优化了计算效率。实验表明,该方法在多数基准上表现更优,且优于当前最优方法。通过聚类优化搜索过程,并在消融实验中验证有效性。进一步应用于仇恨言论与偏见检测,关闭发现的神经元后分析其影响,尤其在印度语境下评估偏见,验证了方法的有效性。本研究推动了对神经元级责任的理解,为未来概念神经元的发现与干预提供新路径。
原文摘要 · Abstract (English)
Locating neurons that are responsible for final predictions is important for opening the black-box large language models and understanding the inside mechanisms. Previous studies have tried to find mechanisms that operate at the neuron level but these methods fail to represent a concept and there is also scope for further optimization of compute required. In this paper, with the help of concept vectors, we propose a method for locating significant neurons that are responsible for representing certain concepts and term those neurons as concept neurons. If the number of neurons is n and the number of examples is m, we reduce the number of forward passes required from O(n*m) to just O(n) compared to the previous works and hence optimizing the time and computation required over previous works. We also compare our method with several baselines and previous methods and our results demonstrate better performance than most of the methods and are more optimal when compared to the state-of-the-art method. We, as part of our ablation studies, also try to optimize the search for the concept neurons by involving clustering methods. Finally, we apply our methods to find, turn off the neurons that we find, and analyze its implications in parts of hate speech and bias in LLMs, and we also evaluate our bias part in terms of Indian context. Our methodology, analysis and explanations facilitate understating of neuron-level responsibility for more broader and human-like concepts and also lay a path for future research in this direction of finding concept neurons and intervening them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。