让大模型生成能暴露错误且输出正确的单元测试,提升代码自动调试效果。
Learning to Generate Unit Tests for Automated Debugging
- 基于任务描述训练模型生成带正确预期输出的错误检测用例。
- 相比基线提升7.59%,在MBPP+上使大模型调试准确率提高12.35%。
- 适用于提升LLM代码调试能力,尤其适合高阶开发者和自动化工具
单元测试在评估代码正确性及为大语言模型提供反馈方面至关重要,推动了自动化测试生成的发展。然而,我们发现生成能揭示错误的测试输入与不依赖真实答案正确预测测试输出之间存在权衡。为此,我们提出UTGen,教会大模型根据任务描述生成既能暴露错误又能给出正确预期输出的单元测试。由于模型生成的测试可能引入噪声(如错误预测输出),我们进一步设计UTDebug:(i) 通过测试时计算扩展提升输出预测精度,(ii) 基于多个生成测试进行验证与回溯修改,防止过拟合,帮助大模型有效调试。实验表明,UTGen在衡量错误揭示输入与正确输出共存的指标上优于其他基于LLM的基线7.59%。结合UTDebug后,其反馈使Qwen2.5 32B在HumanEvalFix和我们自建的更难版MBPP+上的pass@1准确率分别提升3.17%和12.35%。此外,使用Qwen2.5 32B生成的测试反馈可使GPT-4o的调试性能提升13.8%。最后,我们证明UTGen作为代码正确性评判者表现更优,在HumanEval+上以best-of-10采样策略下,相较一个最先进的8B奖励模型提升4.43%。
原文摘要 · Abstract (English)
Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to large language models (LLMs), motivating automated test generation. However, we uncover a trade-off between generating unit test inputs that reveal errors when given a faulty code and correctly predicting the unit test output without access to the gold solution. To address this trade-off, we propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs based on task descriptions. Since model-generated tests can provide noisy signals (e.g., from incorrectly predicted outputs), we propose UTDebug that (i) scales UTGen via test-time compute to improve UT output prediction, and (ii) validates and backtracks edits based on multiple generated UTs to avoid overfitting, and helps LLMs debug effectively. We show that UTGen outperforms other LLM-based baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs. When used with UTDebug, we find that feedback from UTGen's unit tests improves pass@1 accuracy of Qwen2.5 32B on HumanEvalFix and our own harder debugging split of MBPP+ by over 3.17% and 12.35% (respectively) over other LLM-based UT generation baselines. Moreover, we observe that feedback from Qwen2.5 32B-based UTGen model can enhance debugging with frontier LLMs like GPT-4o by 13.8%. Lastly, we demonstrate that UTGen is a better judge for code correctness, outperforming a state-of-the-art trained 8B reward model by 4.43% on HumanEval+ with best-of-10 sampling using Qwen2.5 7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。