arXiv:2510.14444cs.LGcs.AI2025-10被引 1

通过局部重建,无需大规模重训练即可高效压缩大模型且保持性能。

A Free Lunch in LLM Compression: Revisiting Retraining after Pruning

  • 逐子模块局部重建激活,替代全模型重训练
  • 仅需1/10计算量和数据,720亿参数模型仍达相近精度
  • 重建窗口大小影响小,适合内存受限场景

训练后剪枝可大幅降低大语言模型推理开销,但通常会损害性能,除非保留权重被调整。由于全模型重训练在大模型规模下成本高昂,近期工作多聚焦于更复杂的剪枝准则以选择更优稀疏模式而无需适应。本文通过局部重建重新审视这一权衡:剪枝后,逐个子集在校准集上微调模型参数,使其匹配原始密集模型的中间激活。我们在多个模型家族与规模(最高达720亿参数)上评估该方法,发现三个核心结论:第一,局部重建是有效的适应机制,性能接近全模型重训练,但所需数据与计算量减少一个数量级以上,即使使用PEFT技术也如此;第二,重建存在广泛的“免费午餐”区域——只要重建区域包含至少一个非线性子模块,最终性能对窗口大小不敏感,可优先根据内存限制选择粒度;第三,重建降低了剪枝准则的重要性:随着模型规模增大,复杂准则与简单基线间的性能差距缩小,使简单方法重新具备竞争力。总体而言,本研究挑战了“大模型剪枝后适应不可行”的主流观点。

原文摘要 · Abstract (English)

Post-training pruning can substantially reduce LLM inference costs, but it often degrades quality unless the remaining weights are adapted. Since global retraining is expensive at LLM scale, recent work has largely focused on increasingly sophisticated pruning criteria that aim to select better sparsity patterns without adaptation. We revisit this trade-off through local reconstruction: after pruning, we adapt one subset of the model parameters at a time on a calibration set, training it to match the corresponding intermediate activations of the dense model. We evaluate local reconstruction across model families and scales, up to 72B parameters, and establish three main findings. First, local reconstruction is an effective adaptation mechanism for LLMs: it matches post-pruning retraining while using over an order of magnitude less data and compute, even when using PEFT techniques. Second, reconstruction exhibits a broad "free-lunch" regime in granularity, i.e., the reconstruction parameter window: as long as the reconstructed region contains at least a nonlinear submodule, final quality is largely insensitive to the window size, allowing granularity to be chosen primarily based on memory constraints. In contrast, reconstructing individual matrices, despite being the natural approach often proposed in the literature, consistently underperforms, as small matrix-level errors accumulate into larger activation drift. Lastly, reconstruction reduces the relative importance of the pruning criterion: performance gaps between sophisticated criteria and simple baselines shrink with model scale, making simple methods competitive again. Overall, our results challenge the prevailing view that post-pruning adaptation is impractical for LLMs.

模型压缩剪枝轻量化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。