用大模型优化测试脚本生成,解决代码动态变化难题。
Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models
- 基于案例推理的4R循环系统,动态维护测试案例库。
- 在华为两个产品线实验中,生成准确率提升32%,重复生成减少41%。
- 适合自动化测试团队和大模型应用开发者参考。
本文探索大语言模型(LLMs)在生成功能测试脚本中的潜力,这需要理解目标软件的动态代码结构。为此,我们提出一种基于案例推理(CBR)的系统,采用4R循环(检索、复用、修改、保留),通过维护测试意图描述与对应脚本的案例库,辅助大模型生成测试脚本。为提升用户体验,引入Re4优化方法,包含基于重排序的检索微调和强化复用微调。首先,通过高语义与脚本相似度识别正例,提供可靠的伪标签以微调检索模型,无需昂贵标注;随后进行监督微调与强化学习微调,使大模型更契合实际生产场景,确保案例的准确复用。在华为Datacom的两个产品开发单元上进行的大量实验表明,所提的CBR+Re4方法具有显著优势。特别地,该方法有效缓解了大模型的重复生成问题。
原文摘要 · Abstract (English)
In this work, we explore the potential of large language models (LLMs) for generating functional test scripts, which necessitates understanding the dynamically evolving code structure of the target software. To achieve this, we propose a case-based reasoning (CBR) system utilizing a 4R cycle (i.e., retrieve, reuse, revise, and retain), which maintains and leverages a case bank of test intent descriptions and corresponding test scripts to facilitate LLMs for test script generation. To improve user experience further, we introduce Re4, an optimization method for the CBR system, comprising reranking-based retrieval finetuning and reinforced reuse finetuning. Specifically, we first identify positive examples with high semantic and script similarity, providing reliable pseudo-labels for finetuning the retriever model without costly labeling. Then, we apply supervised finetuning, followed by a reinforcement learning finetuning stage, to align LLMs with our production scenarios, ensuring the faithful reuse of retrieved cases. Extensive experimental results on two product development units from Huawei Datacom demonstrate the superiority of the proposed CBR+Re4. Notably, we also show that the proposed Re4 method can help alleviate the repetitive generation issues with LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。