arXiv:2607.02436cs.SEcs.AI2026-07被引 1

提升推理努力比增加工具更能提高代码生成首次成功率。

Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

  • 通过对比不同推理强度和工具使用,发现核心问题是推理不足。
  • 高推理努力使首次完美运行率从28%升至89%,纠错提示减少五倍。
  • 工具虽增加42%~68%成本但未提升功能或可靠性,适合关注视觉设计者。

本研究通过90次独立的智能体运行,评估了多种能力对代码生成质量的影响。所有运行基于同一详细规格构建实时回溯看板应用,采用14项功能评分标准(满分42分)与视觉质量评审。实验涵盖多个模型版本、两种代理框架、两种推理强度、测试工具及两种设计导向提示。结果显示:模型能力层级主导表现,前沿模型接近满分,本地模型仅得24至37分。分析表明,容器部署是主要失败点,首次尝试失败率达44%,且随模型迭代显著变化,而总分波动不足1分。测试工具使成本上升42%~68%,但未改善功能得分或可靠性,即使在界面可见性上亦无提升。将推理强度从高提升至超高,使首次完美运行率从28%增至89%,纠错提示减少约五倍,成本增加9%~29%。设计导向提示使视觉质量从3.0升至4.5(5分制),而一言概括其指令即可复现全部效果。结论:多数首次失败源于推理薄弱,应通过更强模型或更高推理投入解决,而非依赖检查工具。

原文摘要 · Abstract (English)

Agentic coding assistants are increasingly given extra capabilities, such as browser based testing tools and design oriented system prompts, on the assumption that more capability yields better software. This study tested that assumption directly. Ninety independent agent runs built the same application, a real time retrospective board, from one detailed specification, each scored on a fixed 14 criterion functional rubric (42 point maximum) and a visual quality review. The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. A criterion level analysis revealed what run totals conceal. Container deployment was the dominant defect, failing first try in 44 percent of runs, with its failure rate shifting sharply across model generations while mean totals moved less than a point. The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. The practical lesson is to match the fix to the failure: most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch.

代码生成智能体推理优化评估研究

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。