用多智能体迭代调试，让大模型自动生成代码不再因小错失败。
Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
- 设计多智能体框架SEIDR，通过生成、执行、调试、修复循环改进代码。
- 在PSB2上，用Llama 3-8B实现84.2%的pass@100，解决162/164个C++问题。
- 适合想提升大模型编程自动生成能力的研究者和开发者。
大语言模型（LLM）进行程序合成时存在“近似谬误”：生成的代码看似正确，但因细微错误导致单元测试失败。为此，本文提出多智能体框架SEIDR（Synthesize, Execute, Instruct, Debug, and Repair），通过迭代生成、执行、调试与修复来克服该问题。研究探索了提示优化、候选程序排序算法（词典选择与锦标赛选择）及修复策略（替换、修复、混合）之间的权衡。在Program Synthesis Benchmark 2（PSB2）上，SEIDR优于仅使用OpenAI Codex或传统遗传编程的方法，在Python中至少解决20题，C++中至少解决18题。在HumanEval-C++上，使用GPT-3.5和Llama 3-8B的SEIDR分别解决了163/164和162/164问题，平均pass@100达84.2%。结果表明，该框架有效缓解了大模型程序合成中的近似谬误。
原文摘要 · Abstract (English)
Program synthesis with Large Language Models (LLMs) suffers from a "near-miss syndrome": the generated code closely resembles a correct solution but fails unit tests due to minor errors. We address this with a multi-agent framework called Synthesize, Execute, Instruct, Debug, and Repair (SEIDR). Effectively applying SEIDR to instruction-tuned LLMs requires determining (a) optimal prompts for LLMs, (b) what ranking algorithm selects the best programs in debugging rounds, and (c) balancing the repair of unsuccessful programs with the generation of new ones. We empirically explore these trade-offs by comparing replace-focused, repair-focused, and hybrid debug strategies. We also evaluate lexicase and tournament selection to rank candidates in each generation. On Program Synthesis Benchmark 2 (PSB2), our framework outperforms both conventional use of OpenAI Codex without a repair phase and traditional genetic programming approaches. SEIDR outperforms the use of an LLM alone, solving 18 problems in C++ and 20 in Python on PSB2 at least once across experiments. To assess generalizability, we employ GPT-3.5 and Llama 3 on the PSB2 and HumanEval-X benchmarks. Although SEIDR with these models does not surpass current state-of-the-art methods on the Python benchmarks, the results on HumanEval-C++ are promising. SEIDR with Llama 3-8B achieves an average pass@100 of 84.2%. Across all SEIDR runs, 163 of 164 problems are solved at least once with GPT-3.5 in HumanEval-C++, and 162 of 164 with the smaller Llama 3-8B. We conclude that SEIDR effectively overcomes the near-miss syndrome in program synthesis with LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。