用LoRA微调开源大模型,让自动生成测试用例效果接近商用模型。
An empirical study of LoRA-based fine-tuning of large language models for automated test case generation
- 用LoRA高效微调大模型,提升自然语言转测试用例能力。
- 微调后8B开源模型性能接近预训练的GPT-4.1,差距大幅缩小。
- 提出基于GPT-4o的自动化评估框架,多维度验证生成质量。
从自然语言需求自动生成测试用例仍是软件工程中的难题,因需求表述模糊且需生成结构化可执行的测试产物。尽管大语言模型(LLM)展现出潜力,但其效果依赖任务适配与高效微调策略。本文开展一项全面的实证研究,探讨参数高效微调方法LoRA在需求驱动测试用例生成中的应用。我们在统一实验流程下评估多个开源与专有模型家族,系统分析了LoRA的关键超参数(秩、缩放因子、丢弃率)对下游性能的影响。提出基于GPT-4o的自动化评估框架,从九个质量维度评估生成结果。实验表明,所有开源模型经LoRA微调后性能显著提升,其中Ministral-8B表现最优;更关键的是,微调后的8B开源模型性能可媲美预训练的GPT-4.1模型,证明参数高效适配的有效性。尽管GPT-4.1仍领先,但专有与开源模型间的性能差距在微调后大幅缩小。研究为自动化测试生成中的模型选择、微调策略与评估方法提供了重要参考,表明结合合理微调,成本更低、可本地部署的开源模型可成为专有系统的可行替代方案。
原文摘要 · Abstract (English)
Automated test case generation from natural language requirements remains a challenging problem in software engineering due to the ambiguity of requirements and the need to produce structured, executable test artifacts. Recent advances in LLMs have shown promise in addressing this task; however, their effectiveness depends on task-specific adaptation and efficient fine-tuning strategies. In this paper, we present a comprehensive empirical study on the use of parameter-efficient fine-tuning, specifically LoRA, for requirement-based test case generation. We evaluate multiple LLM families, including open-source and proprietary models, under a unified experimental pipeline. The study systematically explores the impact of key LoRA hyperparameters, including rank, scaling factor, and dropout, on downstream performance. We propose an automated evaluation framework based on GPT-4o, which assesses generated test cases across nine quality dimensions. Experimental results demonstrate that LoRA-based fine-tuning significantly improves the performance of all open-source models, with Ministral-8B achieving the best results among them. Furthermore, we show that a fine-tuned 8B open-source model can achieve performance comparable to pre-fine-tuned GPT-4.1 models, highlighting the effectiveness of parameter-efficient adaptation. While GPT-4.1 models achieve the highest overall performance, the performance gap between proprietary and open-source models is substantially reduced after fine-tuning. These findings provide important insights into model selection, fine-tuning strategies, and evaluation methods for automated test generation. In particular, they demonstrate that cost-efficient, locally deployable open-source models can serve as viable alternatives to proprietary systems when combined with well-designed fine-tuning approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。