用简单强化学习训练15亿参数大模型，性能超顶尖方法且省一半算力。
JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
- 单阶段固定超参训练，无需多步调度或调参。
- 在九个数学基准上达到54.9%和64.3%平均准确率。
- 方法稳定可复现，适合追求高效基线的研究者。
近期大语言模型的强化学习方法趋向复杂：多阶段训练流程、动态超参调度与课程学习策略。这引发根本问题：复杂性是否必要？我们提出JustRL，一种极简方案——单阶段固定超参训练，在两个15亿参数推理模型上实现业界领先性能（九个数学基准平均准确率分别为54.9%和64.3%），且计算量仅为复杂方法的二分之一。相同超参可跨模型直接迁移，训练过程在4000+步骤中持续平稳提升，无崩溃或平台期，无需人为干预。关键消融实验表明，加入标准技巧如显式长度惩罚和鲁棒验证器反而会压缩探索空间，导致性能下降。结果提示：当前复杂性可能是为本可避免的问题所添加。我们开源模型与代码，为社区建立一个简单而可靠的基准。
原文摘要 · Abstract (English)
Recent advances in reinforcement learning for large language models have converged on increasing complexity: multi-stage training pipelines, dynamic hyperparameter schedules, and curriculum learning strategies. This raises a fundamental question: \textbf{Is this complexity necessary?} We present \textbf{JustRL}, a minimal approach using single-stage training with fixed hyperparameters that achieves state-of-the-art performance on two 1.5B reasoning models (54.9\% and 64.3\% average accuracy across nine mathematical benchmarks) while using 2$\times$ less compute than sophisticated approaches. The same hyperparameters transfer across both models without tuning, and training exhibits smooth, monotonic improvement over 4,000+ steps without the collapses or plateaus that typically motivate interventions. Critically, ablations reveal that adding ``standard tricks'' like explicit length penalties and robust verifiers may degrade performance by collapsing exploration. These results suggest that the field may be adding complexity to solve problems that disappear with a stable, scaled-up baseline. We release our models and code to establish a simple, validated baseline for the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。