提出固定率梯度优先剪枝法，显著提升模型压缩效率与精度保持能力。
FGGP: Fixed-Rate Gradient-First Gradual Pruning
- 先按梯度大小、再按权重大小排序，固定比例选择要剪的参数。
- 在90%~98%稀疏度下，多数情况下超越现有最佳方法，甚至超过原始密集网络表现。
- 适用于对模型压缩精度要求高的场景，尤其适合深度网络结构优化。
近年来，随着深度学习模型规模扩大及计算资源需求增长，神经网络剪枝成为降低资源消耗的重要手段，同时需保持模型精度。在非结构化渐进剪枝中，通过逐步移除单个参数直至达到目标稀疏度，已有研究表明应同时考虑梯度和权重大小。本文强调选择策略与优先级顺序的关键作用，提出梯度优先、权重次之的剪枝机制，并采用固定比率子选策略，优于文献中常用的退火式方法。我们在CIFAR-10数据集上，针对VGG-19和ResNet-50两种骨干网络，在多次随机初始化下，验证了90%、95%、98%三种稀疏度目标，以及初始密集与50%稀疏两种情形。结果表明，所提出的固定率梯度优先渐进剪枝（FGGP）方法在大多数实验设置中优于当前最优方案，甚至在某些情况下超越对应密集网络的上限表现，整体排名最高。
原文摘要 · Abstract (English)
In recent years, the increasing size of deep learning models and their growing demand for computational resources have drawn significant attention to the practice of pruning neural networks, while aiming to preserve their accuracy. In unstructured gradual pruning, which sparsifies a network by gradually removing individual network parameters until a targeted network sparsity is reached, recent works show that both gradient and weight magnitudes should be considered. In this work, we show that such mechanism, e.g., the order of prioritization and selection criteria, is essential. We introduce a gradient-first magnitude-next strategy for choosing the parameters to prune, and show that a fixed-rate subselection criterion between these steps works better, in contrast to the annealing approach in the literature. We validate this on CIFAR-10 dataset, with multiple randomized initializations on both VGG-19 and ResNet-50 network backbones, for pruning targets of 90, 95, and 98% sparsity and for both initially dense and 50% sparse networks. Our proposed fixed-rate gradient-first gradual pruning (FGGP) approach outperforms its state-of-the-art alternatives in most of the above experimental settings, even occasionally surpassing the upperbound of corresponding dense network results, and having the highest ranking across the considered experimental settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。