arXiv:2607.22587cs.AI2026-07

提出三信号剪枝法,高效压缩大模型且保持性能

TriSP: Tri-Signal Structured Pruning for Large Language Models

论文配图:TriSP: Tri-Signal Structured Pruning for Large Language Models
图 1 · 摘自论文原文
  • 融合权重、激活与梯度信号,生成更准确的剪枝评分
  • 在LLaMA-7B上20%剪枝时达到6.80的困惑度,50%剪枝时推理速度提升82%
  • 适合追求高效部署的大模型应用者,尤其关注推理加速

大型语言模型虽在多任务上表现优异,但其参数带来的内存与计算开销限制了实际部署。结构化剪枝通过移除注意力头和MLP神经元等完整结构,生成可在普通硬件上高效运行的小型稠密模型。然而,现有方法或依赖高内存的梯度重要性评估,或使用激活统计代理,无法直接衡量移除对损失的影响。此外,重要性准则与剪后恢复策略之间的交互尚未系统研究。本文提出TriSP(三信号结构化剪枝),将权重幅度按激活范数缩放,并结合一阶梯度敏感性,通过几何平均生成通道级评分,同时捕捉结构与损失敏感性信号。配合自适应层间预算分配与低秩适配(LoRA)恢复,TriSP在所有测试配置中实现最低困惑度与最高零样本准确率:在LLaMA-7B上20%剪枝时达6.80的WikiText-2困惑度;50%剪枝时推理吞吐量提升82%,仍保持良好性能。

原文摘要 · Abstract (English)

Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters. Structured pruning addresses this by removing entire structures such as attention heads and Multi-Layer Perceptron (MLP) neurons to produce smaller dense models that run efficiently on standard hardware. However, existing methods rely on either gradient-based importance estimation, which is memory-prohibitive, or activation-based statistical proxies, which do not directly measure the effect of removal on the loss. Furthermore, the interaction between the importance criterion and the post-pruning recovery strategy has not been systematically studied. We propose TriSP (Tri-Signal Structured Pruning), an importance metric that combines weight magnitude scaled by activation norm with first-order gradient sensitivity via a geometric mean, producing a channel-level score that captures both structural and loss-sensitivity signals. Combined with adaptive per-layer budget allocation and low-rank adaptation (LoRA) recovery, TriSP achieves the lowest perplexity and highest zero-shot accuracy across all tested configurations, reaching 6.80 WikiText-2 perplexity at 20% pruning on LLaMA-7B. Inference throughput improves by 82% at 50% pruning, while still maintaining competitive performance.

大模型压缩结构剪枝推理加速LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。