通过神经元动态相似性聚合压缩模型,兼顾精度与规模。
Neural Network Compression by Approximate Differential Equivalence

- 基于神经元功能行为相似性聚类,而非逐权重剪枝。
- 在多种数据集上实现显著参数压缩,精度保持稳定。
- 适合追求模型轻量化且重视理论依据的研究者。
神经网络压缩通常通过基于局部重要性评分(如权重大小)的剪枝实现。本文提出一种互补方法:通过聚合具有相似功能行为的神经元来压缩模型,而非独立移除权重。该方法将训练好的网络建模为多项式微分方程系统,并采用近似前向微分等价(Approximate Forward Differential Equivalence)的凝聚方法,识别出诱导动力学近似一致的神经元。单一容差参数 $\varepsilon$ 控制压缩程度,实现模型大小与预测精度之间的平滑权衡。我们在基于非线性动力系统生成的合成数据集及公开回归基准上评估该方法。在两种设置下,所提方法均实现显著参数减少,同时保持精度,且在相似压缩率下持续优于基于权重大小的剪枝和 Wanda 方法。结果表明,基于微分等价的聚合是一种原理严谨且高效的替代传统以权重为中心剪枝的方案。
原文摘要 · Abstract (English)
Neural network compression is commonly achieved by pruning parameters based on local importance scores, e.g., magnitude-based pruning. We propose a complementary approach that compresses models by aggregating neurons with similar functional behavior rather than removing weights independently. Our method encodes a trained network as a polynomial ODE system and applies a lumping method called Approximate Forward Differential Equivalence to identify neurons with approximately matching induced dynamics. A single tolerance parameter, $\varepsilon$, controls the compression level and induces a smooth trade-off between model size and predictive accuracy. We evaluate the method on synthetic datasets derived from nonlinear dynamical systems with known ground-truth behavior and on public regression benchmarks. Across both settings, the proposed approach achieves substantial parameter reduction while preserving accuracy, and consistently compares favorably with magnitude-based pruning and Wanda at similar compression levels. These results suggest that differential equivalence-based aggregation is a principled and effective alternative to conventional weight-centric pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。