用五种策略生成海量高质量代码合成数据，提升模型语义理解能力。
CodeAlchemy: Synthetic Code Rewriting at Scale
- 设计五种策略生成带语义的合成代码数据，覆盖开发任务与执行轨迹。
- 生成超5000亿词元合成数据，执行追踪覆盖130万+文件、14种语言。
- 新基准测试显示大模型在代码执行预测上仍存巨大差距，适合训练推理型模型者参考。
在原始代码上预训练虽能学习语法，但难以捕捉多样化的现实任务模式。尽管合成数据已显著推动语言模型发展，代码领域仍鲜有突破。本文提出CodeAlchemy，一种大规模合成数据生成框架，通过五种策略将公开代码转化为语义丰富的训练数据：CodeEnhance（质量感知重写）、CodeQA（模板化问题）、CodeDev（开发者任务）、CodeDialogue（多轮对话）和CodeTrace（执行轨迹）。我们在15种语言的三个语料库上处理，生成超过5000亿词元的合成数据及3500亿推理词元，数量级远超以往工作。CodeTrace对130万+文件进行仪器化执行，覆盖14种语言和5000个库，记录控制流、状态变化与库知识。我们引入DevEval（开发者任务）与TraceEval（执行预测）评测基准；前沿模型如Claude Sonnet 4.5在TraceEval上仅达5.6%精确匹配率，暴露出语义理解的重大缺陷。我们的30亿参数模型在HumanEval上达83.5%，MBPP上达63.2%，DevEval胜率8.09%，TraceEval ROUGE-2为15.36，超越10倍更大的前沿模型（如270亿参数Gemma-3与320亿参数Granite-4.0）。
原文摘要 · Abstract (English)
Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats. While synthetic data has proven transformative for language models, code remains largely unexplored beyond limited quality improvements. We present CodeAlchemy, a synthetic data generation framework that transforms publicly sourced code into semantically-rich training data through 5 strategies: CodeEnhance (quality-aware rewriting), CodeQA (template-based problems), CodeDev (developer tasks), CodeDialogue (multi-turn conversations), and CodeTrace (execution traces). We process 3 corpora across 15 languages to generate 500B+ tokens of synthetic data plus 350B reasoning tokens, orders of magnitude more than prior efforts. CodeTrace instruments and executes 1.3M+ files across 14 languages and 5K libraries, capturing control flow, state tracking, and library knowledge. We introduce DevEval (developer tasks) and TraceEval (execution prediction) benchmarks; frontier models like Claude Sonnet 4.5 achieve only 5.6% exact match on TraceEval, revealing critical gaps in semantic understanding. Our 3B models achieve 83.5% on HumanEval, 63.2% on MBPP, 8.09% win rate on DevEval, and 15.36 ROUGE-2 on TraceEval, outperforming frontier models 10x the size including 27B Gemma-3 and 32B Granite-4.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。