层剪枝会严重损害大模型的推理能力,即使微调也难恢复。
On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
- 通过剪枝压缩模型,但生成式推理能力显著下降。
- 在分类任务上可恢复90%性能,推理任务恢复率仍很低。
- 适合关注模型压缩与推理能力平衡的研究者。
近期研究发现,层剪枝可在不进行微调或仅少量微调的情况下有效压缩大语言模型(LLM),并保持分类任务上的优异表现。然而,生成式推理任务如GSM8K和HumanEval⁺却表现出明显性能退化。我们发现,剪枝不仅导致文本表面质量下降,更造成关键算法能力丧失,包括算术计算和括号匹配等。在仅使用单个80GB GPU、无预训练规模数据与算力的现实约束下,采用基于自生成回复的监督微调策略,可在分类任务上恢复至基线性能的90%,但生成式推理性能恢复依然有限。即便在任务对齐的微调设置中——将剪枝模型完全在自生成的GSM8K样本上微调,性能差距依然存在,表明退化并非仅由通用指令数据或参数高效微调引起。进一步分析显示,即使经过近1000亿次后剪枝训练的深度剪枝模型,在简单算术任务上仍存在缺陷,且无需多步生成。总体而言,本文揭示了层剪枝在生成式推理中的实际恢复极限,并为受限后训练条件下模型深度缩减的有效性提供了指导。
原文摘要 · Abstract (English)
Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning. In contrast, generative reasoning tasks, such as GSM8K and HumanEval\textsuperscript{+}, exhibit substantially weaker recovery. We show that beyond surface-level text degradation, pruning leads to a loss of key algorithmic capabilities, including arithmetic computation and balanced parenthesis generation. Under realistic post-training constraints, using a single 80GB GPU and without access to pretraining-scale data or compute, we evaluate a simple recovery strategy based on supervised finetuning with self-generated responses. This approach recovers up to 90\% of baseline performance on classification tasks, but recovery for generative reasoning remains limited. We further find that this gap persists even under a favorable task-aligned recovery setting, where pruned models are fully finetuned on self-generated GSM8K responses, suggesting that the degradation is not merely due to generic instruction data or parameter-efficient tuning. As complementary evidence, we analyze a depth-pruned model trained with nearly 100B post-pruning tokens and find that deficits persist even on simple arithmetic tasks that do not require multi-step generation. Overall, we characterize practical recovery limits of layer pruning for generative reasoning and provide guidance on when depth reduction is effective under constrained post-training regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。