用代码模型自动生成训练数据,减少人工干预,提升开源模型性能。
Seed-Coder: Let the Code Model Curate Data for Itself
- 用大模型自身打分筛选代码数据,实现自动化数据构建
- 在代码生成、推理等任务上超越同规模及部分更大模型
- 适合关注代码智能与低人力成本训练的开发者和研究者
大型语言模型(LLM)预训练中的代码数据对代码任务和通用智能提升至关重要。当前开源模型普遍依赖人工构建代码训练数据,如为特定编程语言设计手工过滤规则或使用人工标注数据训练质量过滤器,但这类方法可扩展性差,易受主观偏见影响,且跨语言维护成本高。为此,我们提出Seed-Coder,一系列80亿参数的开源基座、指令微调和推理模型,极大减少人工参与。其代码预训练数据通过以模型为核心的流水线生成,主要由大模型完成代码数据的评分与筛选。指令模型通过监督微调与偏好优化进一步训练,推理模型则采用长链思维(LongCoT)强化学习提升多步代码推理能力。Seed-Coder在代码生成、代码补全、代码编辑、代码推理及软件工程任务中达到同类规模开源模型的最先进水平,甚至超越部分更大模型。
原文摘要 · Abstract (English)
Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。