用大模型做代码修改的无执行评估，精准预测构建结果和可执行性。
Large Language Model Critics for Execution-Free Evaluation of Code Changes
- 设计基于大模型的评议员，无需运行代码即可评估每步修改质量。
- 在SWE-bench上预测构建成功率达84.8%，可执行性预测F1达91.6%。
- 适合对比不同自动化开发流程生成的代码，支持开源复用。
大型语言模型（LLMs）为自动化软件工程任务（如修复漏洞、添加功能）提供了新路径，通过多步骤的基于LLM的代理工作流实现。然而，现有评估指标主要依赖构建状态和日志分析，信息稀疏，难以全面评估代码变更质量。本文设计了基于LLM的评议员，用于生成针对仓库级代码变更的结构化、严谨的中间/步骤级无执行评估代理。关键假设是拥有问题的黄金测试补丁（即参考感知），以评估生成补丁的语义和可执行性。借助黄金测试补丁作为参考，我们对所有编辑位置的可执行性预测达到91.6%的F1分数；综合该结果，可在84.8%的实例中准确预测SWE-bench上的构建状态。特别地，这种聚焦执行的LLM评议员相比其他无参考与有参考方法，性能提升38.9%至72.5%。此外，我们展示了该参考感知框架在比较不同代理工作流生成补丁方面的实用性。最后，我们开源了该项目的工具库，支持其他代理工作流或基准测试的使用，源码见 https://github.com/amazon-science/code-agent-eval。
原文摘要 · Abstract (English)
Large language models (LLMs) offer a promising way forward for automating software engineering tasks, such as bug fixes, feature additions, etc., via multi-step LLM-based agentic workflows. However, existing metrics for evaluating such workflows, mainly build status and occasionally log analysis, are too sparse and limited in providing the information needed to assess the quality of changes made. In this work, we designed LLM-based critics to derive well-structured and rigorous intermediate/step-level, execution-free evaluation proxies for repo-level code changes. Importantly, we assume access to the gold test patch for the problem (i.e., reference-aware) to assess both semantics and executability of generated patches. With the gold test patch as a reference, we predict executability of all editing locations with an F1 score of 91.6%, aggregating which, we can predict the build status in 84.8% of the instances in SWE-bench. In particular, such an execution-focused LLM critic outperforms other reference-free and reference-aware LLM critics by 38.9% to 72.5%. Moreover, we demonstrate the usefulness of such a reference-aware framework in comparing patches generated by different agentic workflows. Finally, we open-source the library developed for this project, which allows further usage for either other agentic workflows or other benchmarks. The source code is available at https://github.com/amazon-science/code-agent-eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。