用大模型生成高质量编程竞赛测试用例，提升评估准确性。
CodeContests+: High-Quality Test Case Generation for Competitive Programming
- 构建基于大模型的智能测试用例生成系统
- 生成172万条提交数据验证，准确率显著提升
- 适合用于训练和评测大模型推理能力的研究者
编程竞赛因其高推理难度和精确的正确性反馈，已成为训练和评估大语言模型（LLM）推理能力的关键任务。然而，尽管问题描述和解答等数据公开丰富，其测试用例往往难以获取。因此，生成高质量测试用例是构建大规模数据集的必要步骤，且直接决定评估的准确性。本文提出一种基于大模型的代理系统，用于生成编程竞赛问题的高质量测试用例。我们将其应用于CodeContests数据集，构建了改进版CodeContests+。通过172万条带有通过/失败标签的提交数据，评估了CodeContests+中测试用例的质量。结果表明，与CodeContests相比，CodeContests+在评估准确性上显著更高，尤其是真阳性率（TPR）明显提升。进一步的大型语言模型强化学习实验也证实，测试用例质量的提升能带来显著优势。
原文摘要 · Abstract (English)
Competitive programming, due to its high reasoning difficulty and precise correctness feedback, has become a key task for both training and evaluating the reasoning capabilities of large language models (LLMs). However, while a large amount of public problem data, such as problem statements and solutions, is available, the test cases of these problems are often difficult to obtain. Therefore, test case generation is a necessary task for building large-scale datasets, and the quality of the test cases directly determines the accuracy of the evaluation. In this paper, we introduce an LLM-based agent system that creates high-quality test cases for competitive programming problems. We apply this system to the CodeContests dataset and propose a new version with improved test cases, named CodeContests+. We evaluated the quality of test cases in CodeContestsPlus. First, we used 1.72 million submissions with pass/fail labels to examine the accuracy of these test cases in evaluation. The results indicated that CodeContests+ achieves significantly higher accuracy than CodeContests, particularly with a notably higher True Positive Rate (TPR). Subsequently, our experiments in LLM Reinforcement Learning (RL) further confirmed that improvements in test case quality yield considerable advantages for RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。