arXiv:2601.09694cs.CLcs.AI2026-01被引 1

用AI智能剪枝大模型,自动保留关键知识路径。

LLMs can Compress LLMs: Adaptive Pruning by Agents

  • 让大模型当助手,动态决定每层剪多少
  • 45%剪枝率下,问答准确率提升56%,知识保留强19倍
  • 无需重训、能自我纠错,适合追求高效压缩的工程师

随着大语言模型持续扩大规模,训练后剪枝成为降低计算成本、同时保持性能的有前景方法。现有方法如SparseGPT和Wanda通过逐层权重重建或激活感知幅度剪枝实现高稀疏度,但依赖统一或手工设计的每层稀疏比例。此外,近期研究发现剪枝后的模型会严重损失事实知识,结构化剪枝方法在事实问答任务中近乎崩溃。本文提出代理引导剪枝,由基础模型充当自适应剪枝代理,在每轮迭代中智能选择剪枝层,同时保护关键知识路径。方法结合Wanda启发的权重-激活指标与梯度重要性得分,归一化为z-score以实现模型无关比较。这些统计量由具备自我反思能力的LLM代理处理,可从过往剪枝结果中学习并迭代优化策略。检查点回滚机制在困惑度下降超过阈值时自动恢复,保障模型质量。我们在Qwen3(4B和8B参数)上评估,约45%稀疏度下,相比结构化剪枝基线:MMLU准确率相对提升56%,FreebaseQA事实知识保留能力提高19倍,困惑度下降69%。框架无需重训,模型无关,仅需2-4次回滚即可完成21-40轮迭代,证明基础模型能有效指导其他基础模型的压缩。

原文摘要 · Abstract (English)

As Large Language Models (LLMs) continue to scale, post-training pruning has emerged as a promising approach to reduce computational costs while preserving performance. Existing methods such as SparseGPT and Wanda achieve high sparsity through layer-wise weight reconstruction or activation-aware magnitude pruning, but rely on uniform or hand-crafted heuristics to determine per-layer sparsity ratios. Moreover, recent work has shown that pruned LLMs suffer from severe factual knowledge degradation, with structured pruning methods experiencing near-total collapse in factual question-answering capabilities. We introduce agent-guided pruning, where a foundation model acts as an adaptive pruning agent to intelligently select which layers to prune at each iteration while preserving critical knowledge pathways. Our method constructs layer-wise sensitivity profiles by combining Wanda-inspired weight-activation metrics with gradient importance scores, normalized as z-scores for model-agnostic comparison. These statistics are processed by an LLM agent equipped with self-reflection capabilities, enabling it to learn from previous pruning outcomes and iteratively refine its strategy. A checkpoint rollback mechanism maintains model quality by reverting when perplexity degradation exceeds a threshold. We evaluate our approach on Qwen3 models (4B and 8B parameters) at approximately 45% sparsity, demonstrating substantial improvements over structured pruning baselines: 56% relative improvement in MMLU accuracy, 19x better factual knowledge retention on FreebaseQA, and 69% lower perplexity degradation. Notably, our framework requires no retraining, operates in a model-agnostic manner, and exhibits effective self-correction with only 2-4 rollbacks across 21-40 iterations, demonstrating that foundation models can effectively guide the compression of other foundation models.

模型剪枝智能压缩大模型优化自纠正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。