提出一种保持信息量的剪枝方法，提升模型压缩效果。
Mutual Information Preserving Neural Network Pruning
- 基于相邻层激活间的互信息选择保留节点
- 剪枝后模型可重训练，性能优于现有方法
- 适用于训练前或训练后剪枝，效果稳定
剪枝已成为降低大模型资源需求的主要手段。自彩票票券假说提出以来，研究多聚焦于初始化时或训练后的剪枝。然而，近期理论发现表明，鲁棒剪枝模型的样本效率与剪枝掩码和训练数据集之间的互信息（MI）成正比，无论剪枝发生在何时。本文由此出发，提出互信息保持剪枝（MIPP），一种适用于训练前或训练后、基于结构化激活的剪枝方法。其核心思想是选择能保留相邻层激活间互信息的节点，从而维持数据与掩码间的互信息。该方法使上游层剪枝后激活可映射至下游层，保证了重训练可行性。实验表明，MIPP在训练前或训练后剪枝中均持续优于现有先进方法。
原文摘要 · Abstract (English)
Pruning has emerged as the primary approach used to limit the resource requirements of large neural networks (NNs). Since the proposal of the lottery ticket hypothesis, researchers have focused either on pruning at initialization or after training. However, recent theoretical findings have shown that the sample efficiency of robust pruned models is proportional to the mutual information (MI) between the pruning masks and the model's training datasets, \textit{whether at initialization or after training}. In this paper, starting from these results, we introduce Mutual Information Preserving Pruning (MIPP), a structured activation-based pruning technique applicable before or after training. The core principle of MIPP is to select nodes in a way that conserves MI shared between the activations of adjacent layers, and consequently between the data and masks. Approaching the pruning problem in this manner means we can prove that there exists a function that can map the pruned upstream layer's activations to the downstream layer's, implying re-trainability. We demonstrate that MIPP consistently outperforms state-of-the-art methods, regardless of whether pruning is performed before or after training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。