量化让代码大模型推理更耗能，剪枝省电但影响性能。
An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
- 测试星火2模型量化与剪枝对能耗和速度的影响
- 量化导致能耗上升，因吞吐量下降且精度略降
- 适合关注模型压缩与能效平衡的开发者
本研究探讨了量化与剪枝策略在降低代码大语言模型（LLMs）推理能耗中的作用。以StarCoder2为实验对象，发现量化虽减小模型体积，却因吞吐量降低导致整体能耗上升，并伴随部分精度损失；而剪枝虽有效减少能耗，但显著损害模型性能。结果揭示了大模型压缩中的核心挑战与权衡。建议未来工作聚焦于面向硬件优化的量化方法，以在保持高精度的同时实现更高能效。
原文摘要 · Abstract (English)
This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy demands with quantization due to lower throughput and some accuracy losses. Conversely, pruning reduces energy usage but impairs performance. The results highlight challenges and trade-offs in LLM model compression. We suggest future work on hardware-optimized quantization to enhance efficiency with minimal loss in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。