提出高效选择代码预训练模型的方法,避免盲目试错。
How to Select Pre-Trained Code Models for Reuse? A Learning Perspective
- 用代理模型评估代码模型的迁移能力,不修改原参数。
- 100个模型中仅需100秒完成筛选,效率提升超百倍。
- 适合想快速复用代码模型的研究者和开发者。
预训练语言模型再微调是代码智能任务(如代码生成、摘要、漏洞检测)的有效方法,但大规模代码预训练成本高昂。目前已有CodeBERT、CodeT5、CodeGen、Code Llama等数百个公开预训练代码模型(PCMs)。如何为特定任务选择最优模型成为关键挑战。本文系统研究了PCMs的可复用性,对比了按大小、训练数据或暴力微调的直观方法,发现其性能差或代价高。为此,提出基于学习的模型选择策略:训练代理模型,通过分析预训练模型隐层特征与任务标签的分布差异,以距离作为迁移能力指标。在涵盖4250万至30亿参数的100个开源模型上实验,新方法将选择时间从2700小时降至100秒,性能损失低于6%,显著提升效率。
原文摘要 · Abstract (English)
Pre-training a language model and then fine-tuning it has shown to be an efficient and effective technique for a wide range of code intelligence tasks, such as code generation, code summarization, and vulnerability detection. However, pretraining language models on a large-scale code corpus is computationally expensive. Fortunately, many off-the-shelf Pre-trained Code Models (PCMs), such as CodeBERT, CodeT5, CodeGen, and Code Llama, have been released publicly. These models acquire general code understanding and generation capability during pretraining, which enhances their performance on downstream code intelligence tasks. With an increasing number of these public pre-trained models, selecting the most suitable one to reuse for a specific task is essential. In this paper, we systematically investigate the reusability of PCMs. We first explore three intuitive model selection methods that select by size, training data, or brute-force fine-tuning. Experimental results show that these straightforward techniques either perform poorly or suffer high costs. Motivated by these findings, we explore learning-based model selection strategies that utilize pre-trained models without altering their parameters. Specifically, we train proxy models to gauge the performance of pre-trained models, and measure the distribution deviation between a model's latent features and the task's labels, using their closeness as an indicator of model transferability. We conduct experiments on 100 widely-used opensource PCMs for code intelligence tasks, with sizes ranging from 42.5 million to 3 billion parameters. The results demonstrate that learning-based selection methods reduce selection time to 100 seconds, compared to 2,700 hours with brute-force fine-tuning, with less than 6% performance degradation across related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。