评测代码生成是否满足任务前提条件，发现模型高分低质。
ContractEval: A Benchmark for Evaluating Contract-Satisfying Assertions in Code Generation
- 构建新基准，显式标注任务前提条件并验证代码是否遵守。
- 五款主流代码大模型在标准提示下合同满足率仅0%。
- 适合关注代码可靠性与安全性的开发者和研究者。
当前代码生成评估仅针对满足所有前置条件的输入测试功能正确性，但任务描述常隐含前提条件，而评估过程会过滤违反条件的输入。这导致生成代码虽在通过率上表现良好，却未遵守实际所需的约束。为此，我们提出ContractEval，一个用于评估生成代码是否满足此类前提条件（即合约）的基准。该基准基于HumanEval+和MBPP+，包含364个任务，每个任务包含三部分：(i) 重构后的任务描述，显式说明合约；(ii) 通过神经符号管道（结合LLM与SMT求解器）生成的测试用例，用于判断代码是否满足合约；(iii) 参考代码及对应的合约。使用ContractEval评估五款代表性开源代码大模型，发现其功能正确性与合约满足性存在巨大差距：在标准提示下，通过率pass@1为75%-82%，但合约满足率为0%；即使在提示中明确写出合约，满足率也仅达23%-41%。结果表明，当前大模型在生成代码时难以满足合约要求，揭示合约满足性是代码生成质量中一个关键且被忽视的维度。代码已开源：https://github.com/suhanmen/ContractEval。
原文摘要 · Abstract (English)
Current code generation evaluation measures functional correctness on well-formed inputs that satisfy all input preconditions. This paradigm has a critical limitation: task descriptions often leave these preconditions implicit, while evaluation filters out inputs that violate them. As a result, generated code may achieve high pass@k scores while failing to enforce the preconditions that the task actually requires. To address this gap, we introduce ContractEval, a benchmark for evaluating whether generated code enforces such preconditions--commonly referred to as contracts. Built on HumanEval+ and MBPP+, ContractEval consists of 364 tasks, each with three components: (i) descriptions reconstructed to explicitly state the contracts, (ii) test cases synthesized through a neuro-symbolic pipeline that pairs an LLM with an SMT solver to evaluate whether generated code satisfies these contracts, and (iii) reference code combined with contracts. Using ContractEval to evaluate five representative open-source code LLMs, we reveal a stark disparity between functional correctness and contract satisfaction. Under standard prompting, these models achieve pass@1 of 75-82% with 0% contract satisfaction. Even when contracts are explicitly stated in the prompt, the satisfaction rate reaches only 23-41%. This indicates that current LLMs struggle to satisfy contracts in their generated code, establishing contract satisfaction as a crucial and previously overlooked axis of code generation quality. Our code is available at https://github.com/suhanmen/ContractEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。