arXiv:2603.07777cs.LGcs.CL2026-03被引 1

针对代码模型长输出难题,提出新强化学习方法提升生成质量与稳定性。

Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models

  • 采用条件截断掩码和动态温度调节,增强长序列生成能力与多样性。
  • 移除高裁剪比例的KL惩罚,使解法多样性显著提升,性能相对基线提高17.6%。
  • 适用于追求高效训练与高质量代码生成的研究者,尤其适合长上下文任务。

现代代码生成模型输出更长、能力增长加速,导致传统训练方法、算法和数据集失效。为此,我们提出MicroCoder-GRPO,一种改进的组相对策略优化方法,包含三项创新:条件截断掩码提升长输出潜力并保持训练稳定,基于多样性的温度选择维持并促进输出多样性,去除高裁剪比例的KL损失以增强解法多样性。MicroCoder-GRPO在LiveCodeBench v6上相对于强基线最高实现17.6%的相对提升,尤其在扩展上下文评估中表现更优。同时,我们发布了MicroCoder-Dataset——一个更具挑战性的训练语料,在300步内比主流数据集带来3倍性能增益;以及MicroCoder-Evaluator——一个评估框架,评估准确率提升约25%,执行速度加快约40%。通过超过三十项受控实验的综合分析,揭示了七个方面共34条训练洞见,证明合理训练的模型可达到大模型的竞争力。

原文摘要 · Abstract (English)

Modern code generation models exhibit longer outputs, accelerated capability growth, and changed training dynamics, rendering traditional training methodologies, algorithms, and datasets ineffective for improving their performance. To address these training bottlenecks, we propose MicroCoder-GRPO, an improved Group Relative Policy Optimization approach with three innovations: conditional truncation masking to improve long output potential while maintaining training stability, diversity-determined temperature selection to maintain and encourage output diversity, and removal of KL loss with high clipping ratios to facilitate solution diversity. MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6, with more pronounced gains under extended context evaluation. Additionally, we release MicroCoder-Dataset, a more challenging training corpus that achieves 3x larger performance gains than mainstream datasets on LiveCodeBench v6 within 300 training steps, and MicroCoder-Evaluator, a robust framework with approximately 25% improved evaluation accuracy and around 40% faster execution. Through comprehensive analysis across more than thirty controlled experiments, we reveal 34 training insights across seven main aspects, demonstrating that properly trained models can achieve competitive performance with larger counterparts.

代码生成强化学习训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。