评测大模型生成算法测试用例的质量与覆盖能力。
Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure
- 构建包含500题10万解的基准集，聚焦故障覆盖与暴露。
- 19个主流大模型在故障覆盖上平均仅达37.6%准确率。
- 适合关注AI辅助测试、代码验证的研究者和开发者。
我们提出TestCase-Eval，一个系统评估大模型生成算法测试用例的新基准。该基准包含来自Codeforces平台的500个算法题和10万条人工编写代码解法，重点评估两个核心任务：（1）故障覆盖，衡量大模型生成的测试集是否能探测多样输入场景并覆盖广泛潜在错误模式；（2）故障暴露，评估大模型能否构造出可揭示特定错误实现的定制化测试输入。我们对19个当前最先进的开源与专有大模型进行了全面评估，揭示了它们在生成有效算法测试用例方面的优势与局限。
原文摘要 · Abstract (English)
We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation. TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform. It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation. We provide a comprehensive assessment of 19 state-of-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。