arXiv:2608.06766cs.LGcs.AI2026-08被引 1

初始权重的缩放方式决定哪个神经元学会关键特征

Hidden Gauge Controls Feature Specialization in ReLU Networks

  • 通过调节神经元初始缩放参数,可主动选择谁学特征
  • 特征专属性能差异达Θ(D²)量级,远超训练速度影响
  • 适合研究神经网络学习机制与可解释性的人参考

训练过程会改变网络预测,并在内部单元间分配任务相关结构。在过参数化的ReLU网络中,多个神经元初始功能相同,但只有一个可能习得教师特征,其余变为冗余。我们称该神经元为特征所有者,探究其是否可通过对初始预测不可见的参数选择来控制。在可解析的高斯教师-学生模型中,固定初始函数,仅改变正齐次缩放规范(gauge)。相反的规范导致不同的特征演化轨迹,且专属性能差异达Θ(D²),无法用全局时间尺度变化解释。在任意数量初始相同的子网络中,仅给一个神经元分配有利规范,即可确定性地使其成为所有者,其余功能贡献降为零。精确的反应-传输分解表明,系数与方向的演化迁移率不同是原因。我们证明了全局选择与功能剪枝,将有限时间选择扩展至可见扰动和小步全批梯度下降,并在群体与有限样本训练中验证了损失、对齐、剪枝与耗散轨迹。因此,初始预测既不能决定特征何时被学习,也不能决定由哪个神经元学习。

原文摘要 · Abstract (English)

Training changes a network's predictions while allocating task-relevant structure across its internal units. In an overparameterized ReLU network, several neurons can begin with exactly the same functional role, yet one may acquire a teacher feature while the others become redundant. We call the identity of that neuron feature ownership and ask whether it can be controlled by a parameter choice invisible to the initial predictor. In a tractable Gaussian teacher--student model, we fix the complete initial function and vary only a positive-homogeneous scaling gauge. Opposite gauges produce distinct feature trajectories and a sharp $Θ(D^2)$ separation in specialization time that no global change of clock can explain. Among any fixed number of initially duplicate students, assigning the favorable gauge to one neuron deterministically selects it as the owner and drives the remaining functional contribution to zero. An exact reaction--transport decomposition attributes the effect to different mobilities for changing a feature's coefficient and direction. We prove global selection and functional pruning, extend finite-time selection to visible perturbations and small-step full-batch gradient descent, and verify the predicted loss, alignment, pruning, and dissipation trajectories in population and finite-sample training. The initial predictor therefore determines neither when the feature is learned nor which neuron learns it.

神经网络特征学习可解释性深度学习理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。