测试生成对代码修复效果影响有限，主要增加成本而非提升结果。
Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
- 分析六种大模型在SWE-bench上的行为，发现写测试频率与任务成败无关。
- 写测试多为打印调试信息，远少于断言式检查，反馈价值较低。
- 调整提示词增减测试数量，最终修复成功率无明显变化。
大型语言模型（LLM）代码代理通过迭代编辑代码、调用工具和验证补丁来解决仓库级问题。在这些流程中，代理常现场编写测试，但其实际价值尚不明确。例如，GPT-5.2几乎不写新测试，却达到顶尖代理的性能水平。这引发核心问题：测试生成是否真正提升问题解决能力，还是仅模仿开发习惯并消耗交互预算？我们分析了六种强模型在SWE-bench Verified上的执行轨迹。结果显示，测试编写普遍存在，但成功与失败任务的测试编写频率相似。当测试被编写时，主要作为观察性反馈渠道，包含价值揭示的打印语句远多于基于断言的检查。通过针对四种模型进行提示干预，分别增加或减少测试生成量，结果表明测试数量的变化并未显著影响最终结果。综合来看，当前代理生成测试的行为更侧重于改变流程与增加开销，而非改善任务成果。
原文摘要 · Abstract (English)
Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。