构建首个可验证代码生成竞赛基准,推动模型生成带证明的正确代码。
VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation

- 基于LeetCode/Codeforces问题构建946个带形式化规格的Rust代码任务
- 模型生成正确代码能力达92%,但证明生成率仅5.3%
- 专为验证代码生成设计,适合研究可信赖AI编程系统
大语言模型能从自然语言生成有用代码,但缺乏正确性保证。可验证代码生成要求模型不仅输出可执行代码,还需提供形式化规格和机器可验证的证明。现有基准多规模小、仅关注流程单一环节、缺少真实证明或严格规格验证,且与主流开发场景脱节。我们提出VeriContest,一个包含946个来自LeetCode和Codeforces的竞赛编程问题的基准,用于Rust语言中使用Verus的可验证代码生成。每个问题均配有专家验证的形式化规格、裁判通过的Rust代码、Verus检查通过的证明,以及正负测试用例。该基准通过三阶段流程构建:从人工验证种子问题出发,经半自动扩展并辅以人工审核。为提升质量,引入测试作为后置条件完备性的质量保障层。VeriContest支持对规格生成、代码生成、证明生成及端到端合成的独立与组合评估。十种前沿模型评估显示:最强模型在自然语言转代码上达92.18%,但在规格生成仅48.31%、证明生成仅13.95%、端到端仅5.29%。结果表明,证明与规格生成是当前模型的核心瓶颈,并确立VeriContest作为衡量与训练未来可信代码生成系统的重要平台。
原文摘要 · Abstract (English)
Large language models can generate useful code from natural language, but their outputs come without correctness guarantees. Verifiable code generation offers a path beyond testing by requiring models to produce not only executable code, but also formal specifications and machine-checkable proofs. Progress in this direction, however, is difficult to measure: existing benchmarks are often small, focus on only one part of the pipeline, lack ground-truth proofs or rigorous specification validation, or target verification settings far from mainstream software development. We present VeriContest, a benchmark of 946 competitive-programming problems from LeetCode and Codeforces for verifiable code generation in Rust with Verus. Each problem pairs a natural language description with expert-validated formal specifications, judge-accepted Rust code, Verus-checked proofs, and positive and negative test suites. VeriContest is constructed through a three-phase pipeline that scales from manually verified seed problems to semi-automated expansion with human-in-the-loop review. To further strengthen benchmark quality, we use testing as an additional quality-assurance layer for validating postcondition completeness. VeriContest supports isolated and compositional evaluation of specification generation, code generation, proof generation, and end-to-end verified program synthesis. Evaluating ten state-of-the-art models reveals a sharp gap between coding ability and verifiable code generation: the strongest model reaches 92.18% on natural-language-to-code generation, but only 48.31% on specification generation, 13.95% on proof generation, and 5.29% end-to-end. These results identify proof and specification generation as the central bottlenecks for models and establish VeriContest as a rigorous platform for measuring and training future systems that generate code with machine-checkable correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。