提出一种融合剪枝与量化的统一方法，提升大模型压缩效果。
AWP: Activation-Aware Weight Pruning and Quantization with Projected Gradient Descent
- 基于投影梯度下降，联合优化剪枝与量化
- 在多个LLM上实现更高精度压缩，优于现有方法
- 理论保证收敛性，适合边缘设备部署
为应对大语言模型（LLMs）的巨大规模，常采用量化和剪枝等模型压缩技术，尤其适用于边缘设备。本文聚焦层间后训练量化与剪枝，将激活感知权重剪枝与稀疏逼近问题关联，并受迭代硬阈值（IHT）成功启发，提出一种通过投影梯度下降实现的激活感知权重剪枝与量化统一方法（AWP）。实验表明，AWP 在多个主流大模型上均优于当前最先进的剪枝与量化方法。同时，本文还提供了该方法在剪枝任务上的理论收敛性保证。
原文摘要 · Abstract (English)
To address the enormous size of Large Language Models (LLMs), model compression methods, such as quantization and pruning, are often deployed, especially on edge devices. In this work, we focus on layer-wise post-training quantization and pruning. Drawing connections between activation-aware weight pruning and sparse approximation problems, and motivated by the success of Iterative Hard Thresholding (IHT), we propose a unified method for Activation-aware Weight pruning and quantization via Projected gradient descent (AWP). Our experiments demonstrate that AWP outperforms state-of-the-art LLM pruning and quantization methods. Theoretical convergence guarantees of the proposed method for pruning are also provided.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。