开源500万代码指令数据集，提升大模型编程能力
OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
- 用合成数据生成+多轮过滤构建大规模代码指令数据
- 在5个基准上显著提升1B至7B模型的编码性能
- 适合研究代码大模型和训练数据构建的开发者
大型语言模型（LLMs）已改变软件开发，实现代码生成、自动调试与复杂推理。但其发展受限于高质量、公开可用的代码任务监督微调（SFT）数据稀缺。为此，我们推出OpenCodeInstruct，目前最大的开源指令微调数据集，包含500万条多样化样本。每条样本包含编程问题、解决方案、测试用例、执行反馈及大模型生成的质量评估。我们使用该数据集对多个基线模型（包括LLaMA和Qwen）进行微调，覆盖1B+、3B+和7B+三种规模。在HumanEval、MBPP、LiveCodeBench和BigCodeBench等主流基准上的全面评估表明，使用OpenCodeInstruct进行SFT后模型性能显著提升。我们还详细介绍了数据构建方法，涵盖种子数据筛选、合成指令与解法生成及过滤流程。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。