通过增加神经元数但不增参数量,提升模型性能。
Expand Neurons, Not Parameters
- 将神经元拆分为更稀疏的子神经元,降低特征间干扰。
- 在布尔任务中准确率显著提升,高干扰场景下效果更明显。
- 适合关注模型可解释性与硬件效率的研究者。
本文证明,在不增加非零参数总数的前提下,通过增加神经元数量可提升网络性能。这一增益源于减少多个特征共享同一神经元导致的干扰。在符号布尔任务中,基于逻辑子句知识将每个神经元拆分为更稀疏的子神经元,系统性降低了多义性度量并提升了任务准确率。值得注意的是,即使随机拆分权重也能近似获得这些收益,表明减少冲突而非精确分配是主要驱动因素。与超叠加假设一致,该框架的优势随干扰程度增加而增强:当多义负载较高时,准确率提升最为显著。将此思想扩展至更真实的模型(包括基于CLIP嵌入的分类器、卷积神经网络及深层多层网络),发现保持非零参数数不变而拓宽网络结构始终能提高准确率。这些结果揭示了一种基于可解释性的宽度利用机制,可在不增加非零参数的情况下缓解超叠加问题,从而提升性能。这一方向与现代加速器特性高度契合——非零参数的数据移动常成为主要瓶颈,而非原始计算。
原文摘要 · Abstract (English)
This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain corresponds with a decrease in interference between multiple features that would otherwise share the same neurons. On symbolic Boolean tasks, splitting each neuron into sparser sub-neurons with knowledge of the clauses systematically reduces polysemanticity metrics and yields higher task accuracy. Notably, even random splits of neuron weights approximate these gains, indicating that reduced collisions, not precise assignment, are a primary driver. Consistent with the superposition hypothesis, the benefits of this framework grow with increasing interference: when polysemantic load is high, accuracy improvements are the largest. Transferring these insights to more realistic models, including classifiers over CLIP embeddings, convolutional neural networks, and deeper multilayer networks, we find that widening networks while maintaining a constant non-zero parameter count consistently increases accuracy. These results identify an interpretability-grounded mechanism to leverage width against superposition, improving performance without increasing the number of non-zero parameters. Such a direction is well matched to modern accelerators, where memory movement of non-zero parameters, rather than raw compute, is often a dominant bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。