提出新剪枝停止准则,显著减少训练时间。
Information Consistent Pruning: How to Efficiently Search for Sparse Networks?
- 基于层间信息与梯度流监测设计停止条件
- 无需中间重训即可保持最终性能
- 适用于多种模型与数据集,效率更优
迭代幅度剪枝方法(IMPs)在减少过参数化深度神经网络中不重要节点方面表现优异,受到广泛关注。然而现有IMPs算法在每轮剪枝中需大量训练时间,存在根本局限。本文提出一种新型停止准则——信息一致性剪枝(InfCoP),通过监测网络各层间的信息与梯度流动来优化剪枝过程,避免在中间步骤重新训练至原始性能,同时保证最终模型性能。实验表明,该方法在多个数据集-模型组合下均优于当前主流IMPs,显著提升效率。我们还提供了算法核心思想的理论分析与基于流的IMPs数学解释。代码已公开于 https://github.com/Sekeh-Lab/InfCoP。
原文摘要 · Abstract (English)
Iterative magnitude pruning methods (IMPs), proven to be successful in reducing the number of insignificant nodes in over-parameterized deep neural networks (DNNs), have been getting an enormous amount of attention with the rapid deployment of DNNs into cutting-edge technologies with computation and memory constraints. Despite IMPs popularity in pruning networks, a fundamental limitation of existing IMP algorithms is the significant training time required for each pruning iteration. Our paper introduces a novel \textit{stopping criterion} for IMPs that monitors information and gradient flows between networks layers and minimizes the training time. Information Consistent Pruning (\ourmethod{}) eliminates the need to retrain the network to its original performance during intermediate steps while maintaining overall performance at the end of the pruning process. Through our experiments, we demonstrate that our algorithm is more efficient than current IMPs across multiple dataset-DNN combinations. We also provide theoretical insights into the core idea of our algorithm alongside mathematical explanations of flow-based IMP. Our code is available at \url{https://github.com/Sekeh-Lab/InfCoP}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。