通过分阶段数据优化与强化学习，提升代码生成模型在竞赛编程中的表现。
DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation
- 先用通用数据微调，再分两阶段强化学习：扩熵防重复，高难题精炼。
- 在LeetCode和Codeforces上达到同规模模型最佳水平，接近顶尖系统。
- 适合关注竞赛编程、强化学习数据设计的研究者与开发者。
近期以推理为核心的模型（如OpenAI o1、DeepSeek R1）推动了可验证奖励强化学习（RLVR）的复兴。然而进展主要集中在数学领域（如AIME），竞赛编程代码生成仍被忽视，且数据构建远不如算法设计受重视。本文研究如何构建适用于RLVR的竞赛编程数据集（即强化学习提示），并提出有效训练策略，在该任务上实现优异表现。方法流程包括：首先对强开源模型进行监督微调（SFT），融合通用与推理密集型数据；随后采用两阶段强化学习，使用可执行、基于测试用例的奖励机制：第一阶段在大量均匀分布的竞赛题目上训练，使用8次采样/提示、较短生成窗口（SFT为32k，此阶段为24k），以扩大熵空间并缓解重复与截断问题；第二阶段采用预-GRPO：在少量高质量难题集上，以64次采样/提示和硬聚焦课程持续保留最困难实例进行更新。我们在Qwen2.5-32B上实现该方法，并在LeetCode及Codeforces周赛上评估，避免数据泄露。结果模型在相似规模中达最优，性能接近DeepSeek v3.1与Doubao-1.5-Thinking。我们还分析了缩放趋势，观察到内部大规模MoE模型在强化学习上具有显著扩展性。本研究总结出适用于竞赛编程代码生成的强化学习数据构建、熵扩展与课程设计的最佳实践。
原文摘要 · Abstract (English)
Recent reasoning-first models (e.g., OpenAI o1, DeepSeek R1) have spurred a resurgence of interest in RLVR. Nevertheless, advances are dominated by mathematics (e.g., AIME), with competitive-programming code generation underexplored and data curation receiving less attention than RL algorithm design. We investigate how to construct RLVR datasets (i.e., RL prompts) and present practical training techniques that yield strong performance on competitive-programming code generation. Our pipeline begins with supervised fine-tuning (SFT) distilled from strong open-source models, augmented with general-purpose and reasoning-intensive data. RL then follows a two-stage process with executable, testcase-driven rewards: first, training on a large, uniformly distributed set of competitive-programming problems using Group Relative Policy Optimization (GRPO) with 8 rollouts per prompt and a relatively short response-generation window (e.g., 32k during SFT and 24k in this stage) to expand entropy and mitigate repetition and truncation; second, we perform \textbf{Pre-GRPO}: updating on a small, high-quality set of challenging problems with a large rollout budget (64 rollouts per prompt) under a hard-focus curriculum that continuously retains the most difficult instances throughout training. We implement our method on Qwen2.5-32B and evaluate on LeetCode and Codeforces weekly contests to avoid data leakage. The resulting model achieves state-of-the-art performance among models of similar scale and is comparable to leading systems such as DeepSeek v3.1 and Doubao-1.5-Thinking. We also examine scaling trends and observe strong RL scaling on an internal large-scale MoE model. Our study distills concise best practices for data curation, entropy expansion, and curriculum design in RLVR for competitive-programming code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。