大模型中的参数模块性由训练数据粒度决定，特定领域模块可被识别且移除会显著影响对应任务。
Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models
- 通过因果分析发现，仅在语言与模态层级存在高度选择性的神经元集群。
- 0.65%-1.14%的神经元具有60%以上域特异性，损伤矩阵对角线比高达595:1。
- 模块性强弱随模型规模增长，适合研究模型可解释性与高效压缩方法的人参考。
大型语言模型是否包含领域特定的参数壳层——即集中、因果必要且移除后仅影响特定领域的神经元群体？我们采用统一因果方法，在两个领域粒度、三种模型家族（1.5B至7B参数）和八个领域上进行测试。在学术主题层面，939,008个组合前馈神经元中无一超过60%域特异性，因果损伤矩阵平坦，尽管领域身份可线性解码至85%以上准确率。在语言与模态层面，0.65%–1.14%的神经元超过60%选择性，损伤矩阵近乎完美对角化（对角线比达595:1），壳层神经元集合基本不重叠（交并比<0.003）。屏蔽代码选择性神经元使数学推理准确率下降16–24个百分点，屏蔽西班牙语或中文神经元则维持在随机水平。壳层强度随模型规模单调上升，且空间交错分布，无法实现组级选择性量化。参数壳层仅在训练数据于词元层面具备模块性时形成。
原文摘要 · Abstract (English)
Do large language models contain domain-specific parametric shells: concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others? We apply a uniform causal methodology across two domain granularities, three model families (1.5B to 7B parameters), and eight domains. At the academic subject level, zero neurons exceed 60\% domain selectivity across 939,008 combined FFN neurons and causal damage matrices are flat, despite domain identity being linearly decodable above 85\% accuracy. At the language and modality level, 0.65--1.14\% of neurons exceed 60\% selectivity, damage matrices are near-perfectly diagonal (ratios up to 595:1), and shell neuron sets are essentially disjoint (IoU $< 0.003$). Masking code-selective neurons reduces mathematical reasoning accuracy by 16--24 percentage points across all models; masking Spanish or Chinese neurons leaves it at or below random. Shell strength increases monotonically with scale and shells are spatially interleaved in a pattern that precludes group-level selective quantization. Parametric shells form where and only where training data was modular at the token level.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。