用高质量数据蒸馏出更强编码模型，效果超越强化学习训练。
OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
- 构建新监督微调数据集，聚焦指令多样性与代码执行过滤。
- 模型仅用SFT即达LiveCodeBench 61.8%、CodeContests 24.6%准确率。
- 开源数据与模型，推动可复现的编程推理研究。
自推理型大语言模型出现以来，通过将推理能力蒸馏到学生模型中，显著缩小了推理模型与普通LLM在编程任务上的差距。然而，多数数据蒸馏进展仍受限于专有数据集或缺乏数据构建、筛选与训练细节。为此，我们构建了一个更优的监督微调（SFT）数据集，用于实现不同规模模型在编码任务上的最先进表现。所蒸馏模型仅使用SFT，便在LiveCodeBench上达到61.8%、CodeContests上达24.6%，超越依赖强化学习训练的替代方案。我们分析了数据来源、代码执行过滤的影响及指令/解法多样性的重要性，发现执行过滤反而降低基准准确率，因此更重视指令多样性而非解法正确性。此外，还考察了模型的令牌效率与推理模式。我们将公开这些数据集和蒸馏模型以供社区使用。
原文摘要 · Abstract (English)
Since the advent of reasoning-based large language models, many have found great success from distilling reasoning capabilities into student models. Such techniques have significantly bridged the gap between reasoning and standard LLMs on coding tasks. Despite this, much of the progress on distilling reasoning models remains locked behind proprietary datasets or lacks details on data curation, filtering and subsequent training. To address this, we construct a superior supervised fine-tuning (SFT) dataset that we use to achieve state-of-the-art coding capability results in models of various sizes. Our distilled models use only SFT to achieve 61.8% on LiveCodeBench and 24.6% on CodeContests, surpassing alternatives trained with reinforcement learning. We then perform analysis on the data sources used to construct our dataset, the impact of code execution filtering, and the importance of instruction/solution diversity. We observe that execution filtering negatively affected benchmark accuracy, leading us to prioritize instruction diversity over solution correctness. Finally, we also analyze the token efficiency and reasoning patterns utilized by these models. We will open-source these datasets and distilled models to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。