通过强化学习与并行思维，显著提升编程竞赛中的推理能力。
Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming
- 用强化学习优化推理过程，提升生成质量。
- 16线程16轮并行生成，平均每题仅需760万词元即达顶尖水平。
- 适合需要长链推理的复杂任务研究者参考。
我们研究了如何通过两种互补方法扩展编程竞赛中的推理词元预算：训练时的强化学习（RL）和测试时的并行思维。在强化学习训练中，观察到验证准确率与生成推理词元的平均数呈近似对数线性关系；通过验证强化学习预热可提升起始点，随机裁剪则使趋势更陡峭。由于全注意力机制下单次生成推理规模快速变贵，我们引入多轮并行思维流水线，将词元预算分配至多个线程与生成、验证、精炼的多轮迭代中。模型在此流水线上端到端训练，以匹配测试结构。从Seed-OSS-36B出发，16线程、每线程16轮的系统，在平均每题使用760万词元的情况下，达到底层强化学习模型的基准通过率（pass@16），并在456道AetherCode难题上超越GPT-5-high。
原文摘要 · Abstract (English)
We study how to scale reasoning token budgets for competitive programming through two complementary approaches: training-time reinforcement learning (RL) and test-time parallel thinking. During RL training, we observe an approximately log-linear relationship between validation accuracy and the average number of generated reasoning tokens over successive checkpoints, and show two ways to shift this training trajectory: verification RL warmup raises the starting point, while randomized clipping produces a steeper trend in the observed regime. As scaling single-generation reasoning during RL quickly becomes expensive under full attention, we introduce a multi-round parallel thinking pipeline that distributes the token budget across threads and rounds of generation, verification, and refinement. We train the model end-to-end on this pipeline to match the training objective to the test-time structure. Starting from Seed-OSS-36B, the full system with 16 threads and 16 rounds per thread matches the underlying RL model's oracle pass@16 at pass@1 using 7.6 million tokens per problem on average, and surpasses GPT-5-high on 456 hard competitive programming problems from AetherCode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。