arXiv:2411.04905cs.CLcs.PL2024-11ACL被引 139

OpenCoder开源顶级代码模型,含全流程可复现的训练数据与方法。

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

  • 提供可复现的数据处理流程与完整训练协议
  • 性能媲美顶尖闭源模型,关键在于数据清洗与合成数据优化
  • 适合科研人员复现、改进代码生成模型

代码大模型在代码生成、推理任务和智能体系统中日益重要。尽管开源代码大模型性能逐步逼近闭源模型,但具备可复现数据处理流程与透明训练协议的高质量模型仍稀缺。这主要受资源、伦理及竞争优势制约。为此,我们推出OpenCoder——一个性能达顶尖水平的代码大模型,并作为研究社区的“开放食谱”。不同于以往工作,我们不仅发布模型权重与推理代码,还开放训练数据、完整数据处理流程、严格的消融实验结果与详细训练协议,支持开放科学。通过该释放,我们识别出构建顶级代码模型的关键要素:(1)代码优化的数据清洗规则与去重方法;(2)相关文本语料的召回;(3)在渐进式与监督微调阶段使用高质量合成数据。此举旨在全面开放顶级代码模型的各个环节,使OpenCoder既可直接使用,也可作为研究基础,推动代码AI的可复现进步。

原文摘要 · Abstract (English)

Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems. While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.

代码生成开源模型可复现LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。