首次评测AI编程代理的工程规范性，发现流程严谨性直接影响代码质量。
RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents
- 从规划、验证、恢复等五方面量化AI编程过程纪律
- 流程规范的代理代码正确率提升17%，过程评分平均高41%
- 适合关注AI开发可靠性与可复现性的工程师和研究者
智能体编程通过引入技能、超能力与工程纪律等机制，日益用于增强底层大模型以完成真实软件工程任务。现有评测几乎仅关注结果正确性：生成代码是否通过测试或解决问题。我们指出，仅看结果不够：通过盲目试错、无计划、无验证或缺乏容错而达成正确解的代理，本质上不如遵循良好工程规范的代理可靠。为此，我们提出RigorBench，首个专门衡量AI编程代理过程纪律的基准。该基准涵盖五大维度：规划一致性、验证覆盖率、恢复效率、放弃质量及原子动作完整性，通过加权求和生成综合得分（RigorScore）。我们设计30个任务，覆盖五大类别——先规划再构建、必须验证、死循环陷阱、知进退、不破坏构建，并在控制实验中对比主流框架有无工程纪律的效果。结果表明，结构化流程使过程评分平均提升41%，下游结果正确率提高17%，首次提供定量证据证明‘如何编码’与‘编码什么’同等重要。我们开源完整基准、评分标准与轨迹分析工具。
原文摘要 · Abstract (English)
Agentic coding harnesses - such as Agent-Skills, Superpowers, and Agent-Rigor - are increasingly deployed to augment underlying LLMs for real-world software engineering tasks. Existing benchmarks evaluate these agents almost exclusively on outcome correctness: whether generated code passes tests or resolves issues. We argue that this outcome-only lens is insufficient: an agent that arrives at a correct solution through reckless trial-and-error, without planning, verification, or graceful recovery, is fundamentally less reliable than one that follows sound engineering discipline. We introduce RigorBench, the first benchmark designed to measure process discipline in AI coding agents. RigorBench evaluates these harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. A composite RigorScore aggregates these dimensions into a single metric via a weighted sum. We curate a suite of 30 tasks spanning five categories - Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don't Break the Build-and evaluate leading harnesses in a controlled with/without experimental design against baseline coding assistants. Our results show that structured process discipline not only improves process quality scores by an average of 41% but also raises downstream outcome correctness by 17%, providing the first quantitative evidence that how agents code matters as much as what they produce. We release the full benchmark, scoring rubrics, and trajectory analysis tools as open-source artifacts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。