评估代码代理生成的项目是否便于后续代理理解与维护。
BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases

- 设计构建-查找协议,分离代码正确性与可读性评价
- 在高先验任务中恢复准确率接近饱和,重点比对检查成本
- 适合关注代码可维护性与协作效率的研究者
多数代码代理评测关注生成代码是否行为正确。这仍很重要,但越来越多的工程工作由代理管理:一个代理创建代码库,后续代理则在其上进行检查、审计或扩展。此时,代码库不仅是任务的答案,更是未来工作的沟通载体。即使强代理几乎满足可见行为目标,代码库在表达意图和设计决策方面仍可能存在差异。本文提出BUILD-AND-FIND协议,评估下游代理从生成代码库中恢复原始设计意图的能力及所需检查成本。每个任务包含一个构建者(见隐藏规范)和一个查找者(仅见代码库与带追溯的多选题集)。该协议区分行为正确性与文档可恢复性,报告恢复准确率、重复性、实现覆盖率与检查努力度。准确率与稳定性作为门槛:仅当恢复稳定成功时,努力度才具意义。在相同意图可恢复的产物间,较低检查努力意味着更易定位意图。问题仅控与规范仅控分别量化通用先验与规范访问影响;审计机制区分遗漏主张与查找失败,并验证正确答案是否引用代码证据。在发布的高先验任务包中,恢复准确率接近饱和,因此检查努力与查找者特异性效应成为主要对比依据。
原文摘要 · Abstract (English)
Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce BUILD-AND-FIND, a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires. For each task, a builder sees a hidden repository specification and creates a codebase; a finder sees only the codebase and a specification-traced multiple-choice question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably. Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate. Question-only and spec-only controls quantify generic priors and specification access, while audits separate omitted claims from finder failures and check whether correct answers cite artifact evidence. In the released high-prior task pack, recovery accuracy is near saturation, so inspection effort and finder-specific effects provide the main panel-local comparison.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。