压缩模型能省电,但只有特定方法有效。
Energy Considerations for Large Pretrained Neural Networks
- 测试9个模型,用三种压缩法对比耗电量。
- 隐写容量压缩让耗电大幅下降,其他方法效果不明显。
- 适合关注模型环保性的研究者和开发者。
日益复杂的神经网络架构取得了惊人性能,但其庞大的计算资源需求消耗大量电力,凸显了潜在的环境影响。以往研究发现大型预训练模型存在显著冗余,但多数工作聚焦于压缩后性能保持,对电力消耗的直接影响关注较少。本文量化了未压缩与压缩模型的能耗,考察压缩作为节能手段的有效性。实验涵盖9个参数量从800万到1.38亿不等的预训练模型。首先在无压缩条件下训练各模型,记录耗电量、训练时间等数据;随后应用三种压缩技术:隐写容量减少、剪枝、低秩分解。在每种压缩后重新测量耗电量、训练时间与模型精度。结果表明,剪枝和低秩分解在能耗及其他指标上均无显著改善,而隐写容量减少几乎在所有情况下都带来显著节能效果。研究讨论了这些发现的意义。
原文摘要 · Abstract (English)
Increasingly complex neural network architectures have achieved phenomenal performance. However, these complex models require massive computational resources that consume substantial amounts of electricity, which highlights the potential environmental impact of such models. Previous studies have demonstrated that substantial redundancies exist in large pre-trained models. However, previous work has primarily focused on compressing models while retaining comparable model performance, and the direct impact on electricity consumption appears to have received relatively little attention. By quantifying the energy usage associated with both uncompressed and compressed models, we investigate compression as a means of reducing electricity consumption. We consider nine different pre-trained models, ranging in size from 8M parameters to 138M parameters. To establish a baseline, we first train each model without compression and record the electricity usage and time required during training, along with other relevant statistics. We then apply three compression techniques: Steganographic capacity reduction, pruning, and low-rank factorization. In each of the resulting cases, we again measure the electricity usage, training time, model accuracy, and so on. We find that pruning and low-rank factorization offer no significant improvements with respect to energy usage or other related statistics, while steganographic capacity reduction provides major benefits in almost every case. We discuss the significance of these findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。