高稀疏率大模型剪枝中,多源校准能更好平衡不同能力的保留。
Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
- 用多源数据混合校准,提升稀疏模型在多种任务上的表现
- 在60%稀疏度下,多源校准比单一来源高出51.9分代码能力
- 适合追求高稀疏下多能力均衡的大模型剪枝研究者
在后训练大模型剪枝中,校准数据常被视为次要细节,因平均评估显示影响较小。我们发现此结论是平均效应的误判:当稀疏度达60%时,仅2.85分的常识准确率差异,对应代码保留能力相差51.9分。对15个数据源的能力分解分析表明,校准困惑度与通用能力正相关,但与数学或代码能力负相关,无单一数据源在所有能力上均最优。这促使提出能力均衡的多源校准策略。在相同校准预算下,基于真实数据的混合策略在LLaMA-3.1-8B上超越所有单源,优于C4 18.8分;优势随稀疏度上升,在LLaMA-3.1-70B上仍成立。由于先进模型预训练数据通常不可用,我们进一步提出信息引导自校准(IGSP),仅需基础模型和评估分类体系,生成分能力样本池并选取低冗余样本,性能优于Self-Cal和SGS达4.8分。结果表明,校准应视为能力覆盖问题,多源设计是高稀疏剪枝中保留能力的关键原则。
原文摘要 · Abstract (English)
Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects. We show that this conclusion is an averaging artifact: at 60\% SparseGPT sparsity, calibration strategies separated by only 2.85 points in averaged commonsense accuracy differ by 51.9 points in Code retention. Across 15 sources, capability-decomposed analysis reveals an opposing pattern: calibration perplexity is positively associated with General retention but negatively associated with Math or Code retention, leaving no evaluated single source uniformly strong across capabilities. This finding motivates capability-balanced multi-source calibration. Under the same calibration budget, a balanced real-data mixture outperforms every evaluated single source on LLaMA-3.1-8B, beating C4 by 18.8 points; the advantage grows with sparsity and persists on LLaMA-3.1-70B. Because the original pretraining data of advanced LLMs are often inaccessible, we further introduce Information-Guided Self-Calibration for Pruning (IGSP). Using only the base model and evaluation taxonomy, IGSP generates capability-stratified pools and selects low-redundancy samples within capability-specific perplexity ranges, outperforming Self-Cal and SGS by up to 4.8 points. Together, these results recast calibration as a capability-coverage problem and identify multi-source design as a practical principle for preserving capabilities in high-sparsity LLM pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。