通过博弈论发现Transformer MLP层中神经元的协作模式,揭示其功能结构。
Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs
- 用博弈论建模神经元协作,识别具有非加性效应的稳定组合。
- 在多个模型上发现协作组合的协同增益高于基线聚类方法。
- 适合研究模型内部机制、可解释性与功能模块的学者参考。
微调的大语言模型编码了丰富的任务特定特征,但这些表征在MLP层中的具体形式仍不清晰。对LoRA更新的实证分析显示,新特征集中在中间层的MLP中,但这些层规模庞大,难以揭示有意义的结构。先前的探测研究提示统计先验可能随深度增强、分裂或消失,因此需要研究神经元如何协同工作而非孤立存在。本文提出一种基于联盟博弈理论的机械可解释性框架,将神经元视为参与享乐博弈的代理,其偏好反映其对局部计算的协同贡献。利用最响应效用和PAC-Top-Cover算法,我们提取出稳定的神经元联盟:联合消融时产生非加性影响的组。随后追踪它们在各层间的演化:持续存在、分裂、合并或消失。应用于在标量信息检索任务上微调的LLaMA、Mistral和Pythia重排序器,该方法发现的联盟协同增益始终高于聚类基线。通过揭示神经元如何协作编码特征,享乐联盟揭示了超越解耦的高阶结构,生成了在跨领域中功能重要、可解释且具预测性的计算单元。
原文摘要 · Abstract (English)
Fine-tuned Large Language Models (LLMs) encode rich task-specific features, but the form of these representations, especially within MLP layers, remains unclear. Empirical inspection of LoRA updates shows that new features concentrate in mid-layer MLPs, yet the scale of these layers obscures meaningful structure. Prior probing suggests that statistical priors may strengthen, split, or vanish across depth, motivating the need to study how neurons work together rather than in isolation. We introduce a mechanistic interpretability framework based on coalitional game theory, where neurons mimic agents in a hedonic game whose preferences capture their synergistic contributions to layer-local computations. Using top-responsive utilities and the PAC-Top-Cover algorithm, we extract stable coalitions of neurons: groups whose joint ablation has non-additive effects. We then track their transitions across layers as persistence, splitting, merging, or disappearance. Applied to LLaMA, Mistral, and Pythia rerankers fine-tuned on scalar IR tasks, our method finds coalitions with consistently higher synergy than clustering baselines. By revealing how neurons cooperate to encode features, hedonic coalitions uncover higher-order structure beyond disentanglement and yield computational units that are functionally important, interpretable, and predictive across domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。