让大模型自己生成测试用例来调试代码，提升编程能力。
Revisit Self-Debugging with Self-Generated Tests for Code Generation
- 提出执行后与执行中两种自我调试范式
- 执行中调试能更好克服自生成测试的偏差
- 适合需要高可靠性代码生成的场景
大语言模型在代码生成方面取得显著进展，但仍难以应对超出基础能力的任务。近期提出的自调试机制通过利用测试执行反馈来提升代码生成性能，但真实场景中高质量测试用例有限。因此，自动生成测试的自调试成为有前景的解决方案，但其局限性和实际潜力尚未充分探索。本文针对多样化编程问题，提出两种自调试范式：执行后与执行中自调试。在自包含的Python编程任务中发现，执行后自调试在基础问题上表现不佳，但在竞争性任务中有改进潜力，原因在于自生成测试引入了偏差；而执行中自调试通过仅利用执行过程中的中间状态，有效缓解了该偏差，从而提升代码生成质量。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown significant advancements in code generation, but still face challenges on tasks beyond their basic capabilities. Recently, the notion of self-debugging has been proposed to boost the performance of code generation by leveraging execution feedback from tests. Despite its promise, the availability of high-quality tests in real-world scenarios is limited. In this context, self-debugging with self-generated tests is a promising solution but lacks a full exploration of its limitations and practical potential. Therefore, we investigate its efficacy on diverse programming problems. To deepen our understanding, we propose two distinct paradigms for the process: post-execution and in-execution self-debugging. Within the scope of self-contained Python programming tasks, we find that post-execution self-debugging struggles on basic problems but shows potential for improvement on competitive ones, due to the bias introduced by self-generated tests. On the other hand, in-execution self-debugging enables LLMs to mitigate the bias by solely leveraging intermediate states during execution, thereby enhancing code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。