让大模型多次尝试修正代码错误，显著提升准确率。
How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks
- 通过反馈执行错误结果，让模型迭代修正代码。
- 最多可提升准确率30个百分点，首轮修复效果最明显。
- 80亿参数模型仅靠提示词就能成功自修，适合工程实践参考。
大型语言模型常在首次生成代码时出错，但多数评测仍采用单次尝试设置。本文在七种不同规模和架构的模型上研究了迭代自修复机制（将运行错误反馈给模型进行修正），覆盖Llama 3.1 8B、Llama 3.3 70B、Llama 4 Scout（MoE，16专家）、Llama 4 Maverick（MoE，128专家）、Qwen3 32B、Gemini 2.5 Flash与Gemini 2.5 Pro。在HumanEval（164题）和MBPP Sanitized（257题）上，最多允许五次尝试，自修复普遍提升通过率：HumanEval提升4.9至17.1个百分点，MBPP提升16.0至30.0个百分点。Gemini 2.5 Flash最终通过率最高，达96.3%（HumanEval）和93.8%（MBPP）。大部分改进集中在前两次尝试。错误类型分析显示，断言错误（逻辑错误）修复率仅约45%，而语法和命名错误修复率更高，印证大模型自修正能力的局限性。此前研究表明弱模型需微调才能自修复，本文表明现代指令微调模型仅靠提示即可成功，甚至80亿参数模型也有效。首次对比了密集型与MoE架构在自修复中的表现，并将修复与重采样权衡扩展至现代模型。提示词消融实验发现，思维链式修复比最小提示额外提升最高5.5个百分点（以修复增量衡量）。
原文摘要 · Abstract (English)
Large language models frequently fail to produce correct code on their first attempt, yet most benchmarks evaluate them in a single-shot setting. We investigate iterative self-repair (feeding execution errors back to the model for correction) across seven models spanning three families and both open-weight and proprietary providers: Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B, Gemini 2.5 Flash, and Gemini 2.5 Pro. On HumanEval (164 problems) and MBPP Sanitized (257 problems) with up to five attempts, self-repair universally improves pass rates: +4.9 to +17.1 pp on HumanEval and +16.0 to +30.0 pp on MBPP. Gemini 2.5 Flash achieves the highest final pass rates (96.3% HumanEval, 93.8% MBPP). Most gains concentrate in the first two rounds.Error-type analysis shows assertion errors (logical mistakes) are the hardest to repair at ~45%, while syntax and name errors are repaired at substantially higher rates, connecting to broader findings on the limits of LLM self-correction. Prior work found that weaker models fail at self-repair or require fine-tuning; we show that modern instruction-tuned models succeed with prompting alone, even at 8B scale. We also provide the first comparison of dense and MoE architectures for self-repair, and extend the repair-vs-resampling tradeoff analysis to modern models. A prompt ablation reveals chain-of-thought repair yields up to +5.5 pp additional self-repair gain (measured as improvement in repair delta) over minimal prompting for capable models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。