arXiv:2607.28871cs.SEcs.AI2026-07

测试通过不等于真修复,一半以上验证结果无效。

Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

  • 通过重放测试用例,区分验证结果的真实价值
  • 46%的通过测试无法识别真正缺陷,23.8%的修复完全依赖无效证据
  • 返回错误状态重放结果可提升验证质量,适合改进代码修复系统

当修复代理运行测试并通过时,该结果被视为对报告缺陷的证据。我们测量这种处理是否合理。BSG-VA(故障态/候选态/黄金修复验证分析)在精确的工作树状态下捕获每个验证命令,提取仅含测试的补丁,并在原始有缺陷代码(B)、候选状态(S)和开发者黄金修复(G)上重放。捕获结果与重放结果共同赋予每个事件一个证据角色,从黄金一致的缺陷区分性到回归仅存甚至误导性。在110个任务、643次运行中的3,730个事件中,46.0%的正向可比事件不携带缺陷区分信息;23.8%的基线运行最终修补完全依赖此类无效证据。三臂实验检验将B重放结果反馈给代理是否改变这一模式。缺陷对比反馈相比注意力匹配提醒降低了7.8个百分点的无效闭合率(p=0.0029),并提升了7.4个百分点的缺陷区分证据(p=0.011),修复成功率无显著影响。两项估计均低于预设10个百分点的最小感兴趣效应,实际意义仍不确定。约三分之一改善来自提醒本身;在两次探索性复现中,仅在gpt-5.6-sol模型的非约束工具使用循环下,重放内容带来可检测增量提升。

原文摘要 · Abstract (English)

When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment.

代码修复验证证据大模型软件质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。