通过分析神经元在层间结构变化中的谱参与度,识别冗余神经元。
Spectral structural distortion reveals redundant neurons in neural networks

- 用图谱方法衡量神经元对层间结构变换的贡献。
- 低参与度神经元可被移除且不影响模型性能。
- 适用于传统网络与Transformer,适合模型压缩研究者。
过参数化的神经网络通常包含大量可移除的神经元,但其冗余性成因仍不明确。现有剪枝准则多依赖权重大小、激活强度等局部指标,难以揭示神经元在层变换中的结构作用。本文提出基于层间表示变换引发的谱结构畸变来刻画冗余性:记录每层前/后激活状态,将神经元建模为图节点,构建输入侧与输出侧的关系图,定义谱结构重要性得分以衡量神经元对主导图谱畸变的贡献。低参与度神经元被视为结构冗余,在迭代剪枝中逐次移除,期间不更新参数;达到目标压缩率后仅进行一次微调恢复。在经典网络、编码器型Transformer和解码器型语言模型上的消融实验表明,该方法能有效识别可剪枝神经元与注意力单元,压缩后保持任务性能。结果表明,神经元冗余不仅源于小权重或弱激活,更可通过其在层间关系结构谱畸变中的弱参与度理解。
原文摘要 · Abstract (English)
Overparameterized neural networks often contain many removable neurons, yet what makes a neuron redundant remains poorly understood. Existing pruning criteria commonly rely on local quantities such as weight magnitude, activation strength, or gradient sensitivity, but these measures provide limited insight into the structural role of a neuron in the transformation performed by a layer. Here we show that neuronal redundancy can be characterized by weak participation in the spectral structural distortion induced by layer-wise representation transformations. For each hidden layer of a trained network, we record pre-activation and post-activation hidden states, model neurons as graph nodes, and construct input-side and output-side graphs that describe neuron-level relational structure before and after the layer transformation. We then define a spectral structural importance score that measures the contribution of each neuron to the dominant graph-spectral distortion between these two relational structures. Low-participation neurons are treated as structurally redundant and removed through an iterative pruning process in which scores are recomputed after each structural change. No parameter updates are performed during intermediate pruning rounds; after the target parameter reduction is reached, a single recovery fine-tuning stage is applied to the compact model. Direct ablation analysis and experiments across conventional neural networks, encoder-only Transformers, and decoder-only language models show that this graph-spectral criterion identifies removable neurons and Transformer units while preserving task performance after compression. These results suggest that neural redundancy is not merely a consequence of small weights or weak activations, but can be understood through weak participation in the spectral distortion of layer-wise relational structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。