用可执行规范评估大模型对程序行为的理解能力
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
- 设计可执行的前置后置条件规范，以真实代码库为基础进行评测
- 模型在仓库级任务中仅20.2%通过率，远低于代码生成表现
- 揭示编码能力强不等于理解程序语义，适合评估模型深层理解
大型语言模型（LLMs）能从自然语言生成代码，但其对程序意图行为的把握程度尚不明确。可执行的行为规范通过前置条件和后置条件定义，为评估这种理解提供了具体手段。然而，现有规范生成研究在评估方法、任务设置和规范表达力方面存在局限。我们提出CodeSpecBench，一个基于执行验证的可执行行为规范生成基准。该基准支持函数级与仓库级任务，将规范编码为可执行的Python函数。数据源自多样化的实际代码库，能够真实评估正确性（接受有效行为）与完整性（拒绝无效行为）。在CodeSpecBench上评估15个前沿大模型，发现其在仓库级任务中性能急剧下降，最佳模型通过率仅为20.2%。进一步表明，规范生成比代码生成更具挑战性，强大编码能力并不等同于对程序语义的深层理解。数据与代码已开源：https://github.com/SparksofAGI/CodeSpecBench。
原文摘要 · Abstract (English)
Large language models (LLMs) can generate code from natural language, but the extent to which they capture intended program behavior remains unclear. Executable behavioral specifications, defined via preconditions and postconditions, provide a concrete means to assess such understanding. However, existing work on specification generation is constrained in evaluation methodology, task settings, and specification expressiveness. We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol. CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions. Constructed from diverse real-world codebases, it enables a realistic assessment of both correctness (accepting valid behaviors) and completeness (rejecting invalid behaviors). Evaluating 15 state-of-the-art LLMs on CodeSpecBench, we observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate. We further find that specification generation is substantially more challenging than code generation, indicating that strong coding performance does not necessarily reflect deep understanding of intended program semantics. Our data and code are available at https://github.com/SparksofAGI/CodeSpecBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。