用自动生成的单元测试引导代码合成,提升生成代码质量与多样性。
UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance
- 通过模型生成单元测试指导代码生成与验证
- 构建50万+可验证程序数据集,显著提升模型性能
- 适合需要高质量代码训练数据的研究者和开发者
大语言模型在多项任务中表现卓越,但代码生成仍面临挑战。现有方法主要依赖大规模预训练数据或基于提示工程的指令数据合成,前者存在质量不一致问题,后者则受限于指令多样性与模型固有偏差。为此,我们提出UnitCoder,一个系统化流程,利用模型生成的单元测试来引导并验证代码生成过程。结合预训练语料库中的大规模包级检索,生成包含多样化API调用的50万+可验证程序数据集。在多个Python基准测试(BigCodeBench、HumanEval、MBPP)上的评估表明,使用该合成数据微调的模型表现出持续性能提升。特别地,Llama3.1-8B和InternLM2.5-7B在BigCodeBench上的成功率分别从31%和28%提升至40%和39%。本工作展示了一种可扩展的方法,利用模型生成的单元测试从预训练语料中合成高质量代码数据,具有大规模生成多样化后训练数据的潜力。所有代码与数据将公开发布。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet code generation remains a major challenge. Current approaches for obtaining high-quality code data primarily focus on (i) collecting large-scale pre-training data and (ii) synthesizing instruction data through prompt engineering with powerful models. While pre-training data faces quality consistency issues, instruction-based synthesis suffers from limited instruction diversity and inherent biases of LLMs. To address this gap, we introduce UnitCoder, a systematic pipeline leveraging model-generated unit tests to both guide and validate the code generation process. Combined with large-scale package-based retrieval from pre-training corpus, we generate a dataset of 500K+ verifiable programs containing diverse API calls. Evaluations on multiple Python benchmarks (BigCodeBench, HumanEval, MBPP) demonstrate that models fine-tuned on our synthetic data exhibit consistent performance improvements. Notably, Llama3.1-8B and InternLM2.5-7B improve from 31\% and 28\% to 40\% and 39\% success rates on BigCodeBench, respectively. Our work presents a scalable approach that leverages model-generated unit tests to guide the synthesis of high-quality code data from pre-training corpora, demonstrating the potential for producing diverse and high-quality post-training data at scale. All code and data will be released (https://github.com).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。