统一局部与全局反馈,实现高效可扩展的大模型稀疏化。
UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMs
- 结合局部重要性评分与全局优化,无需更新权重。
- 支持任意稀疏度的一次性剪枝,保持低困惑度和高准确率。
- 适用于多种硬件约束,适合部署在资源受限场景。
大型语言模型(LLMs)在各类任务中表现优异,但计算和内存开销巨大。剪枝通过引入稀疏性提供了一条可行路径,同时保持架构灵活性。然而,现有方法难以平衡效率与鲁棒性:局部度量方法逐层剪枝,但在高稀疏度下常失效;全局反馈方法虽能保证一致性,却需昂贵的权重更新或受限的半结构化格式。本文提出UniPruning,一种统一的后训练剪枝框架,融合局部显著性评分的快速性与全局协调的稳定性,基于镜面下降优化,且不更新模型权重。该方法利用快速逐层评分与轻量级全局控制器分配单一稀疏预算,支持无结构及半结构化N:M剪枝。经简短校准后,可一次性生成任意稀疏度的剪枝掩码,并自适应硬件约束。在多个预训练LLM家族和标准基准上的实验表明,UniPruning始终达到竞争力或更优的困惑度与零样本准确率。消融研究进一步验证了镜面下降与局部显著性锚定的重要性。总体而言,UniPruning为大规模LLM稀疏化提供了高效、原理清晰且可扩展的解决方案。代码已开源:https://github.com/RainbowQTT/UniPruning。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve strong performance across diverse tasks but face prohibitive computational and memory costs. Pruning offers a promising path by inducing sparsity while preserving architectural flexibility. However, existing methods struggle to balance efficiency and robustness: local metric approaches prune layer by layer but often collapse under high sparsity, whereas global feedback methods enforce consistency at the cost of expensive weight updates or restrictive semi-structured formats. We present UniPruning, a unified post-training pruning framework that combines the speed of local saliency metrics with the stability of global coordination, enabled by a mirror descent based optimization, all without updating model weights. UniPruning leverages fast layer-wise scoring and a lightweight global controller to allocate a single sparsity budget, supporting both unstructured and semi-structured N :M pruning within one framework. After a brief calibration, it can generate pruning masks for arbitrary sparsity levels in one shot, and adapts seamlessly to hardware-aware constraints. Extensive experiments on multiple pretrained LLM families and standard benchmarks show that UniPruning consistently delivers competitive or superior perplexity and zero-shot accuracy. Ablation studies further highlight the importance of mirror descent and local saliency anchoring. Overall, UniPruning provides an efficient, principled, and scalable solution for sparsifying large-scale LLMs. Our code is available at: https://github.com/RainbowQTT/UniPruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。