构建测试用例生成新基准，评估模型在长上下文下的覆盖与验证能力。
CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification
- 基于12个开源项目设计多任务评测，涵盖4k至128k token长上下文。
- 所有模型在复杂任务上得分低于35%，即使提供最优上下文仍表现不佳。
- 支持代码执行验证，适合研究大模型测试生成与覆盖率提升。
软件测试是开发关键环节，但用例生成仍是工程师的重复工作。本文提出基准CLOVER，用于评估模型在特定条件下生成和补全测试用例的能力。任务范围从简单断言补全到跨多个文件覆盖指定代码块的测试编写，基于12个Python仓库，分析845个问题，上下文长度达4k至128k tokens。利用代码测试框架，我们提出一种基于覆盖率信息构建检索上下文的方法。尽管短上下文下模型表现相近，但在16k上下文时差异明显。例如GPT-4o和Claude 3.5能有效利用相关代码片段，但所有模型在复杂任务III上得分均低于35%，即使提供理想上下文亦然，凸显该基准的重要性及模型改进空间。该基准已容器化，支持任务间代码执行，代码、数据与构建方法将公开发布。
原文摘要 · Abstract (English)
Software testing is a critical aspect of software development, yet generating test cases remains a routine task for engineers. This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases under specific conditions. Spanning from simple assertion completions to writing test cases that cover specific code blocks across multiple files, these tasks are based on 12 python repositories, analyzing 845 problems with context lengths ranging from 4k to 128k tokens. Utilizing code testing frameworks, we propose a method to construct retrieval contexts using coverage information. While models exhibit comparable performance with short contexts, notable differences emerge with 16k contexts. Notably, models like GPT-4o and Claude 3.5 can effectively leverage relevant snippets; however, all models score below 35\% on the complex Task III, even with the oracle context provided, underscoring the benchmark's significance and the potential for model improvement. The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。