LEAP实现大模型无结构稀疏化端到端学习，提升推理精度。
LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
- 用伯努利-加温-逻辑斯蒂松弛替代传统参数化，实现可学习的无结构剪枝
- 在0.5B至8B模型上，50%与60%稀疏度下平均零样本准确率提升2.59点
- 适合追求高稀疏度下保持模型性能的研究者和部署工程师
非结构化稀疏性现已被新型GPU内核和数据流硬件原生加速，使瓶颈从推理执行转向剪枝算法。当前最先进的非结构化大语言模型剪枝方法基于逐层代理，源自最优大脑手术原则，但在激进稀疏度下牺牲了端到端精度。端到端替代方案如MaskLLM和PATCH表明可学习掩码能弥合这一差距，但其基于模式的分类参数化随每行有效掩码数量增长而扩展，无法适配无结构设置。我们提出LEAP，以每个权重的伯努利-加温-逻辑斯蒂松弛取代这种不可行的参数化，使端到端无结构掩码学习成为可能。在五个从0.5B到8B参数的大模型家族中，于50%和60%稀疏度下，LEAP相比最佳逐层基线ADMM，在六项任务的平均零样本准确率上提升了+2.59点。
原文摘要 · Abstract (English)
Unstructured sparsity is now natively accelerated by recent GPU kernels and dataflow hardware, shifting the bottleneck from inference execution to the pruning algorithm. State-of-the-art methods for unstructured LLM pruning are layer-wise surrogates derived from the Optimal Brain Surgeon principle, and they sacrifice end-to-end accuracy, especially under aggressive sparsity. End-to-end alternatives such as MaskLLM and PATCH show that learnable masks can close this gap, but their categorical-over-patterns parameterization scales with the number of valid masks per row and does not port to the unstructured setting. We introduce LEAP, which replaces this intractable parameterization with a per-weight Bernoulli-via-Gumbel-sigmoid relaxation that makes end-to-end unstructured mask learning tractable. Across five LLM families from 0.5B to 8B parameters at 50% and 60% sparsity, LEAP improves six-task average zero-shot accuracy by +2.59 points on average over ADMM, the best layer-wise baseline in our sweep.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。