让代码和测试用例一起进化,无需预先定义测试用例。
CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation
- 代码与测试用例协同演化,由自然语言描述生成
- 在多个LLM上超越现有方法,提升代码生成质量
- 适合自动化编程、智能开发工具场景
大型语言模型(LLMs)在自动化代码生成方面表现出色。然而,现有方法通常严重依赖预定义的测试用例,这在缺乏测试用例的场景中变得不切实际。尽管先前工作探索了程序与测试用例间的筛选机制,却忽略了对测试用例的优化。为此,我们提出CoCoEvo——一种基于LLM的协同演化框架,可同时演化程序与测试用例。该框架通过自然语言问题描述和函数声明直接生成程序与测试用例,消除了对预定义测试用例的依赖。框架采用专门设计的进化算子,包括基于LLM的交叉与变异算子用于程序演化,并引入额外的测试用例生成算子用于测试用例演化。此外,我们提出交叉率调度器以平衡探索与收敛,并采用多目标优化方法进行测试用例选择。在多个主流LLM上的实验结果表明,CoCoEvo优于现有方法,在自动化代码生成与测试任务中达到当前最优性能,验证了协同演化技术在自动化编程领域的巨大潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable performance in automated code generation. However, existing approaches often rely heavily on pre-defined test cases, which become impractical in scenarios where such cases are unavailable. While prior works explore filtering techniques between programs and test cases, they overlook the refinement of test cases. To address this limitation, we introduce CoCoEvo, a novel LLM-based co-evolution framework that simultaneously evolves programs and test cases. CoCoEvo eliminates the dependency on pre-defined test cases by generating both programs and test cases directly from natural language problem descriptions and function headers. The framework employs specialized evolutionary operators, including LLM-based crossover and mutation operators for program evolution, along with an additional test case generation operator for test case evolution. Additionally, we propose optimization strategies such as a crossover rate scheduler to balance exploration and convergence, and a multi-objective optimization method for test case selection. Experimental results on multiple state-of-the-art LLMs demonstrate that CoCoEvo surpasses existing methods, achieving state-of-the-art performance in automated code generation and testing. These results underscore the potential of co-evolutionary techniques in advancing the field of automated programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。