收集28万+竞赛编程漏洞测试用例，助力代码模型安全验证
Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforces
- 从Codeforces平台挖掘真实漏洞测试用例，构建可复现的对抗样本集
- 涵盖5,578道题目与2,196个可被攻破的提交代码，支持模型鲁棒性评估
- 适合大模型代码生成、自动化测试及竞赛编程研究者使用
软件在日常生活中广泛应用于关键场景，确保其正确性至关重要。传统方法依赖测试用例评估软件正确性：若测试通过，则认为程序无误；若失败，则暴露缺陷。然而，测试结果的可靠性取决于测试集完整性，存在漏检（即程序通过所有测试但实际含错）的风险。因此，设计诱发错误的测试用例对提升验证效果至关重要。为支持数据驱动的测试用例生成，尤其针对大语言模型生成的代码，本文构建了名为Codehacks的数据集。该数据集来自Codeforces在线评测平台，包含5,578道编程题的288,617个漏洞测试用例（即“黑客攻击”），每题附有自然语言描述，并提供2,196个可被对应攻击破解的提交代码。此数据集可用于训练和评估代码生成模型的鲁棒性。
原文摘要 · Abstract (English)
Software is used in critical applications in our day-to-day life and it is important to ensure its correctness. One popular approach to assess correctness is to evaluate software on tests. If a test fails, it indicates a fault in the software under test; if all tests pass correctly, one may assume that the software is correct. However, the reliability of these results depends on the test suite considered, and there is a risk of false negatives (i.e. software that passes all available tests but contains bugs because some cases are not tested). Therefore, it is important to consider error-inducing test cases when evaluating software. To support data-driven creation of such a test-suite, which is especially of interest for testing software synthesized from large language models, we curate a dataset (Codehacks) of programming problems together with corresponding error-inducing test cases (i.e., "hacks"). This dataset is collected from the wild, in particular, from the Codeforces online judge platform. The dataset comprises 288,617 hacks for 5,578 programming problems, each with a natural language description, as well as the source code for 2,196 submitted solutions to these problems that can be broken with their corresponding hacks. Keywords: competitive programming, language model, dataset
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。