用少量参数微调大模型生成单元测试,效果接近全量微调。
Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study
- 采用LoRA、(IA)^3、提示调优等方法,仅更新部分参数。
- LoRA在多数情况下性能接近全量微调,提示调优成本最低。
- 适合关注训练效率与测试覆盖率的研究者和开发者。
参数高效微调(PEFT)方法通过仅调整模型的部分参数,显著降低大型语言模型(LLMs)微调的计算成本,同时保持性能。现有研究已探索将PEFT与LLMs应用于多种代码任务,但其有效性因任务而异。当前最先进的方法仍依赖全量微调生成单元测试,而PEFT在该任务中的应用尚未充分研究。本文系统比较了全量微调与多种PEFT方法(包括LoRA、(IA)^3、提示调优),覆盖十三种不同架构与规模的模型。基于标准基准数据集,评估了语法正确性、CodeBLEU、pass@1、指令覆盖率、分支覆盖率及突变分数。结果表明,LoRA在多个场景下表现可媲美全量微调;若重视训练成本,提示调优对大模型最为经济。然而,全量或PEFT微调后的模型生成的可执行测试案例较少,因存在调用不存在方法或类型不匹配问题。但对于可执行的测试,其覆盖率优于基线模型。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters, offer a promising solution by reducing the computational costs of tuning large language models (LLMs) while maintaining their performance. Existing studies have explored using PEFT and LLMs for various code-related tasks and found that the effectiveness of PEFT techniques is task-dependent. The state-of-the-art is limited to using LLMs with full fine-tuning to generate unit tests. The application of PEFT techniques in unit test generation remains underexplored. This paper investigates both full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt tuning, across thirteen models of different architectures and sizes. We use well-established benchmark datasets to evaluate their effectiveness in unit test generation and measure syntax correctness, CodeBLEU, pass@1, instruction coverage, branch coverage, and mutation score of the generated tests. Our findings show that LoRA can deliver performance comparable to full fine-tuning for unit test generation in several cases. If training costs are valued, prompt tuning is the most cost-effective approach, particularly for large models. However, the models tuned with full fine-tuning or PEFT may generate fewer executable test cases than the baseline model because they generate more tests calling nonexistent methods or having type mismatches. For the generated ones that are executable, the ones from the tuned models show better test coverage than those from the baseline model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。