arXiv:2502.16638cs.LGcs.AI2025-02CVPR被引 33

自动联合剪枝与量化,让模型更小更快。

Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression

  • 构建量化感知依赖图,自动搜索剪枝空间。
  • 保证每层位宽约束,实现高效联合优化。
  • 适用于各类网络结构,无需手动调参。

结构化剪枝与量化是减少深度神经网络(DNN)规模的核心技术,通常独立应用。联合通过协同优化可生成更小、高质量的模型,但现有方法因工程复杂(多阶段流程)、黑箱优化(需大量超参数调优)及架构泛化能力不足而未被广泛采用。为此,本文提出框架GETA,可自动高效地对任意DNN进行联合结构化剪枝与量化感知训练。GETA引入三项关键创新:(i) 量化感知依赖图(QADG),为通用量化感知DNN构建剪枝搜索空间;(ii) 部分投影随机梯度法,确保逐层位宽约束被满足;(iii) 新的联合学习策略,融合剪枝与量化间的可解释关系。在卷积神经网络与Transformer架构上的数值实验表明,本方法性能优于或相当现有联合剪枝量化方法。

原文摘要 · Abstract (English)

Structured pruning and quantization are fundamental techniques used to reduce the size of deep neural networks (DNNs) and typically are applied independently. Applying these techniques jointly via co-optimization has the potential to produce smaller, high-quality models. However, existing joint schemes are not widely used because of (1) engineering difficulties (complicated multi-stage processes), (2) black-box optimization (extensive hyperparameter tuning to control the overall compression), and (3) insufficient architecture generalization. To address these limitations, we present the framework GETA, which automatically and efficiently performs joint structured pruning and quantization-aware training on any DNNs. GETA introduces three key innovations: (i) a quantization-aware dependency graph (QADG) that constructs a pruning search space for generic quantization-aware DNN, (ii) a partially projected stochastic gradient method that guarantees layerwise bit constraints are satisfied, and (iii) a new joint learning strategy that incorporates interpretable relationships between pruning and quantization. We present numerical experiments on both convolutional neural networks and transformer architectures that show that our approach achieves competitive (often superior) performance compared to existing joint pruning and quantization methods.

模型压缩剪枝量化自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。