通过激活稀疏性实现零训练的双稀疏化，显著降低大模型部署成本。
DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
- 将运行时激活稀疏性视为动态结构化权重稀疏，构建双稀疏计算负载。
- 在1.39倍加速下，准确率比基线高9.17%，优于现有结构化剪枝方法。
- 无需重新训练，适配千亿参数模型，适合追求高效部署的研究与工程人员。
大语言模型（LLMs）性能强大，但因内存和计算开销高难以部署。现有剪枝方法多忽略运行时激活稀疏性。本文将激活稀疏性重解释为动态结构化权重稀疏，提出DuoGPT框架，通过非结构化权重剪枝与激活稀疏性结合，构建双稀疏（spMspV）计算负载。为保持精度，扩展最优大脑压缩（OBC）框架，引入激活感知校准，并以密集模型输出残差作为修正项。进一步优化方案以支持GPU高效执行，可扩展至百亿参数级模型。在LLaMA-2和LLaMA-3上评估显示，相比基线稠密模型，在1.39倍等速加速下，准确率最高提升9.17%，优于当前先进结构化剪枝方法。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) deliver strong performance but are difficult to deploy due to high memory and compute costs. While pruning reduces these demands, most methods ignore activation sparsity observed at runtime. We reinterpret activation sparsity as dynamic structured weight sparsity and propose DuoGPT, a unified framework that constructs dual-sparse (spMspV) workloads by combining unstructured weight pruning with activation sparsity. To preserve accuracy, we extend the Optimal Brain Compression (OBC) framework with activation-aware calibration and introduce output residuals from the dense model as correction terms. We further optimize the solution for efficient GPU execution, enabling scalability to billion-parameter LLMs. Evaluations on LLaMA-2 and LLaMA-3 show that DuoGPT outperforms state-of-the-art structured pruning methods by up to 9.17% accuracy at an iso-speedup of 1.39$\times$ compared to the baseline dense model. Code is available at Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。