针对大模型剪枝难题，提出自生成校准数据策略提升剪枝后推理能力。
Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models
- 用自生成的难且适中长度推理数据做校准，替代通用数据。
- 在DeepSeek-R1-Distill上使剪枝模型推理能力提升10%-13%。
- 适合需要高效部署复杂推理模型的研究者和工程师。
大型推理模型（LRMs）在复杂推理基准上表现优异，但其长链式思维过程带来显著推理开销。剪枝被视为降低计算成本的可行方案。然而，现有研究主要聚焦于大语言模型（LLMs），对LRMs的剪枝仍属空白。本文首次对LRMs剪枝进行实证研究，发现直接套用现有剪枝方法效果不佳。研究表明，使用自生成推理数据进行校准可显著提升剪枝性能。我们进一步分析了推理数据难度与长度对剪枝结果的影响，发现困难且中等长度的自生成数据是理想的校准数据。基于此，提出选择性自生成推理（SSGR）数据构建策略，为剪枝提供有效校准数据。在DeepSeek-R1-Distill模型系列上的实验表明，该策略使剪枝后模型推理能力相比通用方法提升10%-13%。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning benchmarks. However, their long chain-of-thought reasoning processes incur significant inference overhead. Pruning has emerged as a promising approach to reducing computational costs. However, existing efforts have primarily focused on large language models (LLMs), while pruning LRMs remains unexplored. In this work, we conduct the first empirical study on pruning LRMs and show that directly applying existing pruning techniques fails to yield satisfactory results. Our findings indicate that using self-generated reasoning data for calibration can substantially improve pruning performance. We further investigate how the difficulty and length of reasoning data affect pruning outcomes. Our analysis reveals that challenging and moderately long self-generated reasoning data serve as ideal calibration data. Based on these insights, we propose a Selective Self-Generated Reasoning (SSGR) data construction strategy to provide effective calibration data for pruning LRMs. Experimental results on the DeepSeek-R1-Distill model series validate that our strategy improves the reasoning ability of pruned LRMs by 10%-13% compared to general pruning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。