对比三种压缩方法在代码模型中的效率与效果表现。
On the Compression of Language Models for Code: An Empirical Study on CodeBERT
- 实证研究知识蒸馏、量化、剪枝对代码模型的影响。
- 不同任务下压缩效果差异大，无统一最优方案。
- 为开发者选择压缩策略提供依据，平衡速度与精度。
语言模型在软件工程任务中表现优异，但其高昂的计算成本限制了实际应用。为此，研究者开始采用知识蒸馏、量化和剪枝等压缩策略提升代码语言模型的效率，降低推理延迟与内存占用，但常以模型性能下降为代价。然而，当前对这些策略如何影响代码模型的效率与有效性仍缺乏深入理解。本文通过实证研究，评估这三种压缩方法在三类典型任务（漏洞检测、代码摘要、代码搜索）中的表现。结果表明，压缩效果因任务类型和具体方法而异，无普适最优解。研究结论有助于实践者和研究人员根据具体需求，权衡效率与性能，选择最合适的压缩策略。
原文摘要 · Abstract (English)
Language models have proven successful across a wide range of software engineering tasks, but their significant computational costs often hinder their practical adoption. To address this challenge, researchers have begun applying various compression strategies to improve the efficiency of language models for code. These strategies aim to optimize inference latency and memory usage, though often at the cost of reduced model effectiveness. However, there is still a significant gap in understanding how these strategies influence the efficiency and effectiveness of language models for code. Here, we empirically investigate the impact of three well-known compression strategies -- knowledge distillation, quantization, and pruning -- across three different classes of software engineering tasks: vulnerability detection, code summarization, and code search. Our findings reveal that the impact of these strategies varies greatly depending on the task and the specific compression method employed. Practitioners and researchers can use these insights to make informed decisions when selecting the most appropriate compression strategy, balancing both efficiency and effectiveness based on their specific needs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。