arXiv:2605.25939cs.LGcs.AI2026-05

通过正则化让小模型神经元聚焦特定数据原型,提升从权重还原训练数据的能力。

From Latent Space to Training Data: Explainable Specialization in Minimal MLPs

论文配图:From Latent Space to Training Data: Explainable Specialization in Minimal MLPs
图 1 · 摘自论文原文
  • 用覆盖正则约束隐藏层激活,使神经元聚焦于训练样本原型。
  • 覆盖正则在所有数据规模下均实现最低重建误差,优于标准拟合基线。
  • 揭示了排斥性正则需配对吸引项,否则会破坏潜在空间几何结构。

本文研究在最小单隐层MLP中,训练偏差是否可使隐藏神经元实现专业化,并是否有助于从学习到的权重中重构训练数据。针对宽度等于数据集大小的高斯激活MLP,比较三种结构正则:分别鼓励样本覆盖、神经元诱导原型分离,以及隐藏响应低重叠,与标准拟合基线对比。在均匀采样的一维数据集上,从N=3到N=100共480次受控实验中,覆盖正则在所有规模下均实现最低平均重建误差,并提升原型使用专业化比率;分离正则效果混合,重叠惩罚则系统性有害。研究表明该损害非优化失败:重叠活跃方法拟合数据能力相当,但引导优化器进入退化平衡点,使原型中心移出训练输入凸包。覆盖正则无法奖励此逃逸,起到吸引作用;分离正则仅在高温时允许,重叠正则在默认超参下允许。τ扫面与N=100时原型位置可视化验证机制。研究得出简单设计原则:每个排斥性结构损失必须搭配兼容吸引项,否则将导致潜空间几何坍塌。

原文摘要 · Abstract (English)

We here study whether training biases can make hidden neurons specialize in minimal one-hidden-layer MLPs, and whether such specialization improves prototype-based reconstruction of the training dataset from the learned weights. We consider Gaussianactivation MLPs of width equal to dataset size and compare three structural losses that respectively encourage coverage of the training samples, separation between neuron-induced prototypes, and low overlap of hidden responses, against the standard fitting baseline. Experiments on uniformly sampled one-dimensional datasets show a stable pattern from N = 3 to N = 100 across 480 controlled runs. Coverage regularization gives the lowest mean reconstruction error at every tested size and raises the prototype-usage specialization ratio relative to the standard baseline, while separation has mixed effects and overlap penalties are systematically harmful. We show that the harm is not an optimization failure: overlap-active approaches fit the data as well as overlap-free ones but route the optimizer to a degenerate equilibrium in which prototype centers are pushed outside the convex hull of the training inputs. Coverage cannot reward this expulsion and acts as an attractor: separation admits it only at large temperature and overlap admits it at the nominal hyperparameter choice. A direct τ-sweep on the separation-only mask and a prototype-position visualization at N = 100 confirm the mechanism. The findings yield a simple design principle for prototype-recoverability-aware training: every repulsive structural loss must be compensated by a compatible attractor, or it will collapse the latent geometry it was meant to refine.

神经网络原型重构结构正则可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。