用可验证环境动态调整难度，让语言模型强化学习更有效。
RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
- 设计可自动验证的动态难度环境，随模型能力调节挑战性。
- 在400个环境中联合训练，推理能力平均提升3.37%。
- 适合研究强化学习与语言模型结合的学者使用。
我们提出基于可验证环境的强化学习（RLVE），通过程序化生成问题并提供算法可验证的奖励，实现语言模型（LMs）的强化学习规模化。每个可验证环境能根据策略模型的能力动态调整问题难度分布，避免静态数据分布导致的训练信号消失问题。为实现该方法，我们构建了400个精心设计的可验证环境集合——RLVE-Gym。实验表明，环境规模扩展能持续提升模型的泛化推理能力。在所有400个环境上联合训练，使六项推理基准上的平均性能提升3.37%，起点为当前最强的15亿参数推理模型。相比之下，原模型继续训练仅获得0.49%的平均提升，且计算开销超出三倍。代码已公开。
原文摘要 · Abstract (English)
We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。