arXiv:2609.05126cs.LGcond-mat.dis-nn2026-09

用映射熵无监督选出关键神经元,提升模型压缩效率

Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy

论文配图:Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropy
图 1 · 摘自论文原文
  • 基于隐藏层激活统计,用映射熵衡量删减神经元带来的信息损失
  • 在强压缩下,选中的子网络性能优于随机子集,且能保留功能关联结构
  • 适用于无标签场景,适合追求模型轻量化与可解释性的研究者

过参数化神经网络包含远超任务需求的隐藏单元,这引发了一个问题:哪些神经元是关键的?这种区分能否在不依赖标签或梯度的情况下,从表示本身识别?本文将神经元选择视为通过保留部分神经元对隐藏层进行粗粒化的过程,并以映射熵(ME)作为评分标准。该指标衡量因舍弃部分神经元而导致的判别力损失,使ME最小的神经元集合被视为最具信息量。该方法完全无监督,仅依赖隐藏激活的统计特性。在教师-学生网络中,ME优化恢复了最小一致表示,并按隐藏层残余变异性比例保留额外单元;在非线性高斯过程任务中,它选出具有相干功能类映射的神经元,其偏好类别随训练变化。在该任务及经过平移增强的MNIST上,经ME选择的子网络表现优于同等规模的随机子集,尤其在强压缩条件下尤为明显,表明构型可区分性与预测性能直接相关。

原文摘要 · Abstract (English)

Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.

神经元选择无监督学习模型压缩映射熵

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。