用二次规划优化剪枝,大幅提升大模型压缩后的精度。
OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction
- 将每层权重重建转为独立的二次规划问题,共享海森矩阵提升效率。
- 在单张H100上40小时完成80亿参数模型剪枝,峰值内存60GB。
- 适合作为大模型部署前的高效剪枝工具,尤其适合资源受限场景。
后训练剪枝是高效的模型压缩方案,但存在权衡:简单启发式方法虽快却损失精度;而严谨的联合优化方法虽能恢复精度,却难以在现代大模型规模下实现。现有单次剪枝方法如SparseGPT通过近似启发式更新实现可接受的平衡。本文提出OPTIMA,一种实用的单次后训练剪枝方法,兼顾精度与可扩展性。它将层内剪枝后的权重重建建模为独立的行级二次规划(QPs),共享层海森矩阵,使问题高度适用于加速器批量处理。我们实现了一个面向加速器的高效QP求解器,每层仅累积一次海森矩阵,并并行求解多个小型二次规划问题,从而在无需微调的情况下,单加速器实现大规模剪枝。OPTIMA可集成现有掩码选择器,在多个LLM家族和稀疏度设置下一致提升零样本性能,最高达3.97%绝对精度提升。在NVIDIA H100上,80亿参数Transformer模型端到端剪枝耗时40小时,峰值内存60GB。这些结果确立了单次后训练剪枝在精度-效率权衡上的新基准。
原文摘要 · Abstract (English)
Post-training model pruning is a promising solution, yet it faces a trade-off: simple heuristics that zero weights are fast but degrade accuracy, while principled joint optimization methods recover accuracy but are computationally infeasible at modern scale. One-shot methods such as SparseGPT offer a practical trade-off in optimality by applying efficient, approximate heuristic weight updates. To close this gap, we introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability. OPTIMA casts layer-wise weight reconstruction after mask selection as independent, row-wise Quadratic Programs (QPs) that share a common layer Hessian. Solving these QPs yields the per-row globally optimal update with respect to the reconstruction objective given the estimated Hessian. The shared-Hessian structure makes the problem highly amenable to batching on accelerators. We implement an accelerator-friendly QP solver that accumulates one Hessian per layer and solves many small QPs in parallel, enabling one-shot post-training pruning at scale on a single accelerator without fine-tuning. OPTIMA integrates with existing mask selectors and consistently improves zero-shot performance across multiple LLM families and sparsity regimes, yielding up to 3.97% absolute accuracy improvement. On an NVIDIA H100, OPTIMA prunes a 8B-parameter transformer end-to-end in 40 hours with 60GB peak memory. Together, these results set a new state-of-the-art accuracy-efficiency trade-offs for one-shot post-training pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。