arXiv:2601.18091cs.LG2026-01被引 11

针对推理型大模型的剪枝策略需与传统语言模型区别对待。

From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models

  • 区分指令型与推理型模型,采用统一训练分布校准剪枝
  • 深度剪枝适合分类,宽度剪枝更适配生成与长链推理
  • 静态剪枝保留推理能力,动态剪枝在生成任务中表现更优

大语言模型(LLMs)部署成本高昂,推动了模型剪枝研究。然而,现有工作多聚焦于指令跟随型模型,未明确剪枝策略是否适用于显式生成长中间推理链的推理增强型模型。本文对指令型(LLM-instruct)和推理增强型(LLM-think)模型进行了受控剪枝实验,通过匹配原始训练分布进行剪枝校准与恢复数据,实现更稳定可靠的剪枝行为。在17项涵盖分类、生成与推理的任务上评估静态深度剪枝、静态宽度剪枝及动态剪枝。结果表明:深度剪枝在分类任务中表现更优,而宽度剪枝在生成与推理任务中更具鲁棒性;静态剪枝更能保持推理性能,动态剪枝在分类与生成任务中表现突出,但在长链推理中仍具挑战。研究强调需为推理增强型模型设计专门的剪枝策略。代码已公开于https://github.com/EIT-NLP/LRM-Pruning。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly costly to deploy, motivating extensive research on model pruning. However, most existing studies focus on instruction-following LLMs, leaving it unclear whether established pruning strategies transfer to reasoning-augmented models that explicitly generate long intermediate reasoning traces. In this work, we conduct a controlled study of pruning for both instruction-following ($\textbf{LLM-instruct}$) and reasoning-augmented ($\textbf{LLM-think}$) models. To isolate the effects of pruning, we align pruning calibration and post-pruning recovery data with each model's original training distribution, which we show yields more stable and reliable pruning behavior. We evaluate static depth pruning, static width pruning, and dynamic pruning across 17 tasks spanning classification, generation, and reasoning. Our results reveal clear paradigm-dependent differences: depth pruning outperforms width pruning on classification tasks, while width pruning is more robust for generation and reasoning. Moreover, static pruning better preserves reasoning performance, whereas dynamic pruning excels on classification and generation but remains challenging for long-chain reasoning. These findings underscore the need for pruning strategies that explicitly account for the distinct characteristics of reasoning-augmented LLMs. Our code is publicly available at https://github.com/EIT-NLP/LRM-Pruning.

模型剪枝推理增强大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。