用合成数据和课程学习提升代码生成的强化学习效果
A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula
- 教师模型根据学生表现迭代优化问题，生成有难度梯度的合成数据
- 相比单轮生成，有效问题产出率显著提升，支持课程训练
- 适用于大规模语言模型在代码与数学任务上的持续优化
强化学习（RL）已成为超越监督微调提升大语言模型性能的重要范式，但规模化时性能提升受限于数据多样性与结构，而非单纯数量。本文提出一种可扩展的多轮合成数据生成管道：教师模型基于上下文的学生表现摘要，迭代优化问题，无需教师微调即可生成具有结构化难度递进的题目。相比单轮生成，该方法显著提高有效合成问题的产量，并自然产生同一核心任务的更易与更难变体，构成课程训练所需的阶梯。我们在 Llama3.1-8B Instruct 与 Qwen3-8B Base 模型上系统研究了任务难度、课程调度与环境多样性在 RL 训练中的交互影响，还对 Qwen2.5-32B 进行了扩展实验。结果表明，合成数据增强能持续提升域内代码性能，在多数情况下也优于域外数学任务表现，并揭示了课程设计与数据多样性共同塑造 RL 训练动态的实证规律。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a powerful paradigm for improving large language models beyond supervised fine-tuning, yet sustaining performance gains at scale remains an open challenge, as data diversity and structure, rather than volume alone, become the limiting factor. We address this by introducing a scalable multi-turn synthetic data generation pipeline in which a teacher model iteratively refines problems based on in-context student performance summaries, producing structured difficulty progressions without any teacher fine-tuning. Compared to single-turn generation, this multi-turn approach substantially improves the yield of valid synthetic problems and naturally produces stepping stones, i.e. easier and harder variants of the same core task, that support curriculum-based training. We systematically study how task difficulty, curriculum scheduling, and environment diversity interact during RL training across the Llama3.1-8B Instruct and Qwen3-8B Base model families, with additional scaling experiments on Qwen2.5-32B. Our results show that synthetic augmentation consistently improves in-domain code and in most cases out-of-domain math performance, and we provide empirical insights into how curriculum design and data diversity jointly shape RL training dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。