代码修复中循环不等于可靠,需绑定证据与状态
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair

- 提出带证据绑定的状态型修订合约,确保修复过程可验证
- 实验证明旧测试用例使正确修复率下降22.2个百分点
- 适合关注AI代码修复可信性与系统设计的研究者
编码智能体常用的生成-测试-修订循环中,重复并不保证可靠性。本文研究了找到正确补丁与成功保留、验证、提交之间的差距。在30个HumanEval任务上进行5种子实验,共生成900条三次修订轨迹。强制修订下,当前测试用例下的正确率从第一次修订的0.820降至第二次的0.673,但历史上曾正确的比例升至0.847。通过两组相同状态的实验(共2,430个分支)消除处理后风险集偏差。在预设14B模型复现中,过时测试用例导致34/135个正确起点失败,而当前测试用例仅4/135失败,正确率下降22.2点(任务簇95%置信区间[8.9,37.0],精确霍尔姆p=0.0337)。前瞻性540次采样策略虽消除正确起点伤害,但降低了错误起点修复率,未满足联合标准。对24个真实漏洞和四种开发栈的仓库实验显示存在底限效应与组件异质性,但无霍尔姆显著影响。因此,论文分离出准入、保存、接地认证、能力与活性等维度,推导出证据绑定的类型化循环合约,并在参考实现中机械执行其子集:将验证证据绑定到确切代码状态,保留已验证检查点,生成可审计的准入凭证。该实现是可执行规范与符合性产物,不证明修复能力提升或验证器依赖校准。
原文摘要 · Abstract (English)
Generate--test--revise loops are common in coding agents, but repetition alone provides no reliability guarantee. We study the gap between finding a correct patch and retaining, verifying, and submitting it. A sealed five-seed study over 30 HumanEval repairs produces 900 three-revision trajectories. Under forced revision, current correctness with current traces falls from 0.820 after one revision to 0.673 after two, although ever-correct rises to 0.847. Two common-state studies use 2,430 branches from identical frozen programs to remove post-treatment risk-set bias. In a prespecified 14B replication, stale traces harm 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95\% CI $[8.9,37.0]$, exact Holm $p=0.0337$). A prospective 540-rollout policy eliminates observed correct-start harm but reduces wrong-start repair and fails its joint criterion. Repository experiments over 24 bugs and four coder stacks expose floor effects and component heterogeneity without Holm-significant effects. We therefore separate admission, preservation, grounded certification, competence, and liveness. We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts. The implementation is an executable specification and conformance artifact, not evidence of improved repair competence or calibrated verifier dependence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。