用全局门控实现大模型结构化剪枝，省一半参数仍保持性能。
GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models
- 通过投影直通估计器在训练中动态控制剪枝，实时满足预算限制。
- 在LLaMA-2-7B上剪掉50%参数，WikiText-2困惑度仅升至12.18。
- 无需微调，单卡4个周期即可完成，适合资源受限部署场景。
大型语言模型（LLMs）服务成本高，因参数量、注意力计算和键值缓存带来显著内存与延迟开销。本文提出GRASPrune，一种预训练后应用的结构化剪枝框架，联合剪除前馈网络通道与键值头组，在单一全局预算下实现优化。不同于先学习重要性分数再施加预算的方法，GRASPrune使用投影直通估计器学习轻量级门控分数，每步强制执行硬掩码以满足预算，同时保持主干权重冻结。掩码固定后，对保留单元校准缩放因子以缓解剪枝带来的尺度失配，并将这些因子合并进剪裁后的权重，得到无额外参数的小型稠密检查点。在LLaMA-2-7B上，该方法移除50%参数，于WikiText-2上达到12.18困惑度，且在五个基准测试上保持有竞争力的平均零样本准确率，仅需在单张NVIDIA A100 80GB GPU上运行4个周期的512条未标注校准序列，无需全模型微调。
原文摘要 · Abstract (English)
Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a single global budget. Instead of learning importance scores without constraints and applying the budget only after training, GRASPrune learns lightweight gate scores with a projected straight-through estimator that enforces a hard mask satisfying the budget at every step while keeping the backbone weights frozen. After the mask is fixed, we calibrate scaling factors on the retained units to mitigate scale mismatch caused by pruning, and fold these factors into the pruned weights to obtain a smaller dense checkpoint with no extra parameters at inference. On LLaMA-2-7B, GRASPrune removes 50% of parameters and achieves 12.18 perplexity on WikiText-2 while maintaining competitive average zero-shot accuracy on five benchmarks, using four epochs on 512 unlabeled calibration sequences on a single NVIDIA A100 80GB GPU without any full model fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。