arXiv:2507.06920cs.CL2025-07NeurIPS被引 17

用人类+大模型协作生成更全面的代码测试用例,提升评估可靠性。

Rethinking Verification for LLM Code Generation: From Generation to Testing

  • 提出多维指标量化测试用例覆盖度,改进评估体系。
  • SAGA方法检测率达90.62%,验证准确率超LiveCodeBench-v6 10.78%。
  • 适合关注代码生成评估、强化学习奖励设计的研究者。

大语言模型在HumanEval和LiveCodeBench等代码生成基准上表现优异,但这些评测集通常仅包含少量同质化测试用例,导致细微缺陷难以被发现,不仅人为抬高性能指标,还影响基于可验证奖励的强化学习(RLVR)中的奖励估计准确性。为此,本文系统研究了测试用例生成(TCG)任务,提出多维度指标以严格量化测试集完备性;引入人-大模型协同方法SAGA,结合人类编程经验与大模型推理能力,显著提升生成测试用例的覆盖率与质量。同时构建了TCGBench评测基准。实验表明,SAGA在TCGBench上检测率达90.62%,验证准确率为32.58%;由SAGA合成的评测基准验证准确率比LiveCodeBench-v6高出10.78%。结果证明该方法有效,有助于建立可扩展的可靠代码生成评估基础,推动代码生成中RLVR的发展,并为自动化对抗性测试生成与动态基准融合提供支持。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently achieved notable success in code-generation benchmarks such as HumanEval and LiveCodeBench. However, a detailed examination reveals that these evaluation suites often comprise only a limited number of homogeneous test cases, resulting in subtle faults going undetected. This not only artificially inflates measured performance but also compromises accurate reward estimation in reinforcement learning frameworks utilizing verifiable rewards (RLVR). To address these critical shortcomings, we systematically investigate the test-case generation (TCG) task by proposing multi-dimensional metrics designed to rigorously quantify test-suite thoroughness. Furthermore, we introduce a human-LLM collaborative method (SAGA), leveraging human programming expertise with LLM reasoning capability, aimed at significantly enhancing both the coverage and the quality of generated test cases. In addition, we develop a TCGBench to facilitate the study of the TCG task. Experiments show that SAGA achieves a detection rate of 90.62% and a verifier accuracy of 32.58% on TCGBench. The Verifier Accuracy (Verifier Acc) of the code generation evaluation benchmark synthesized by SAGA is 10.78% higher than that of LiveCodeBench-v6. These results demonstrate the effectiveness of our proposed method. We hope this work contributes to building a scalable foundation for reliable LLM code evaluation, further advancing RLVR in code generation, and paving the way for automated adversarial test synthesis and adaptive benchmark integration.

代码生成测试用例LLM评估强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。