用强化学习与搜索提升代码模型的深度思考能力。
o1-Coder: an o1 Replication for Coding
- 结合强化学习与蒙特卡洛树搜索，增强代码生成的推理能力。
- 通过测试用例生成器和迭代微调，实现从伪代码到完整代码的生成。
- 适合关注代码生成与AI深度推理的研究者和开发者。
技术报告介绍了O1-CODER，一个针对编程任务复现OpenAI o1模型的尝试。该框架融合强化学习（RL）与蒙特卡洛树搜索（MCTS），旨在提升模型的System-2思维能力。包含训练测试用例生成器（TCG）以实现标准化代码测试，利用MCTS生成带推理过程的代码数据，并通过迭代微调策略模型，使其先生成伪代码，再输出完整代码。报告还探讨了o1类模型在实际应用中的机遇与挑战，建议向System-2范式过渡，并强调构建世界模型的必要性。后续版本将更新模型进展与实验结果。所有源代码、整理的数据集及衍生模型均公开于https://github.com/ADaM-BJTU/O1-CODER。
原文摘要 · Abstract (English)
The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1 model with a focus on coding tasks. It integrates reinforcement learning (RL) and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking capabilities. The framework includes training a Test Case Generator (TCG) for standardized code testing, using MCTS to generate code data with reasoning processes, and iteratively fine-tuning the policy model to initially produce pseudocode and then generate the full code. The report also addresses the opportunities and challenges in deploying o1-like models in real-world applications, suggesting transitioning to the System-2 paradigm and highlighting the imperative for world model construction. Updated model progress and experimental results will be reported in subsequent versions. All source code, curated datasets, as well as the derived models are disclosed at https://github.com/ADaM-BJTU/O1-CODER .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。