arXiv:2412.15921cs.SEcs.AI2024-12

通过统一剪枝让代码大模型更节能,性能损失仅3%。

Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

  • 同时剪枝词表、层和前馈网络,实现高效压缩。
  • 剪掉22%参数后仍保持97%原性能,微调后更优。
  • 适合需要低耗部署的代码生成应用开发者。

大型语言模型在代码生成任务中广泛应用,但其高计算需求和能耗引发关注。传统结构化剪枝方法针对分类模型设计,难以适配生成式代码大模型产生的高维标记概率序列。现有单一组件剪枝方法也限制了效果。为此,我们提出Flab-Pruner,一种融合词表、层与前馈网络(FFN)的统一剪枝方法,在保持性能的同时显著减少模型参数。此外,我们设计定制化代码指令数据策略,提升剪枝后模型的性能恢复效率。在三个先进代码大模型上对多个生成式代码任务进行评估,结果表明:剪掉22%参数后,模型保留97%原始性能,且经微调后表现相当甚至更优。剪枝模型在存储、显存占用、计算效率及环境影响方面均有显著改善,同时保持良好鲁棒性。本研究为绿色软件工程提供了可持续解决方案,推动大模型在真实代码生成智能应用中的高效部署。

原文摘要 · Abstract (English)

The extensive application of Large Language Models (LLMs) in generative coding tasks has raised concerns due to their high computational demands and energy consumption. Unlike previous structural pruning methods designed for classification models that deal with lowdimensional classification logits, generative Code LLMs produce high-dimensional token logit sequences, making traditional pruning objectives inherently limited. Moreover, existing single component pruning approaches further constrain the effectiveness when applied to generative Code LLMs. In response, we propose Flab-Pruner, an innovative unified structural pruning method that combines vocabulary, layer, and Feed-Forward Network (FFN) pruning. This approach effectively reduces model parameters while maintaining performance. Additionally, we introduce a customized code instruction data strategy for coding tasks to enhance the performance recovery efficiency of the pruned model. Through extensive evaluations on three state-of-the-art Code LLMs across multiple generative coding tasks, the results demonstrate that Flab-Pruner retains 97% of the original performance after pruning 22% of the parameters and achieves the same or even better performance after post-training. The pruned models exhibit significant improvements in storage, GPU usage, computational efficiency, and environmental impact, while maintaining well robustness. Our research provides a sustainable solution for green software engineering and promotes the efficient deployment of LLMs in real-world generative coding intelligence applications.

代码生成模型剪枝绿色AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。