用统计校正提升大模型剪枝效果，不重训练也能高效降本。
Efficient Post-Training Pruning of Large Language Models with Statistical Correction
- 基于权重和激活的一阶统计量校准重要性评分，减少异常值干扰。
- 剪枝后通过解析能量补偿修复分布偏差，性能更稳定。
- 无需反向传播或重训练，适合快速部署在多种大模型上。
后训练剪枝是降低大语言模型（LLMs）规模与推理开销的有效方法，但现有方法常在剪枝质量与计算效率间存在权衡。启发式剪枝方法虽高效，却对激活异常值敏感；基于重构的方法虽能提升保真度，但计算开销巨大。本文提出一种轻量级后训练剪枝框架，基于模型权重与激活的一阶统计特性。剪枝过程中，采用通道级统计量校准基于幅度的重要度评分，降低激活主导通道带来的偏差。剪枝后，通过解析能量补偿修正因权重移除导致的分布失真。两个步骤均无需重训练、梯度或二阶信息。在多个LLM家族、稀疏模式及评估任务上的实验表明，该方法在保持类启发式方法计算成本的同时，显著提升了剪枝性能。结果表明，简单的统计校正可有效用于大语言模型的后训练剪枝。
原文摘要 · Abstract (English)
Post-training pruning is an effective approach for reducing the size and inference cost of large language models (LLMs), but existing methods often face a trade-off between pruning quality and computational efficiency. Heuristic pruning methods are efficient but sensitive to activation outliers, while reconstruction-based approaches improve fidelity at the cost of heavy computation. In this work, we propose a lightweight post-training pruning framework based on first-order statistical properties of model weights and activations. During pruning, channel-wise statistics are used to calibrate magnitude-based importance scores, reducing bias from activation-dominated channels. After pruning, we apply an analytic energy compensation to correct distributional distortions caused by weight removal. Both steps operate without retraining, gradients, or second-order information. Experiments across multiple LLM families, sparsity patterns, and evaluation tasks show that the proposed approach improves pruning performance while maintaining computational cost comparable to heuristic methods. The results suggest that simple statistical corrections can be effective for post-training pruning of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。