arXiv:2605.14392cs.AI2026-05被引 1

模型不再模仿数据,而是自动生成可验证的推理环境来持续提升自身能力。

Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis

论文配图:Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis
图 1 · 摘自论文原文
  • 通过自动生成可执行的Python环境,将训练从数据生成转为环境构建。
  • 在Qwen3-4B上,推理准确率从72.4提升至74.8,相对增长3.3%。
  • 适合研究自进化AI、强化学习与可验证推理的开发者和研究者。

我们探索语言模型自我改进的愿景:模型不只生成待解决的问题或推理轨迹进行模仿,而是主动构建训练环境。在零数据推理强化学习中,这将自我改进从数据生成循环转变为环境构建循环,每个生成物均为可复用的可执行对象,能采样实例、计算参考答案并评分响应。该愿景能否持续进步取决于一个关键特性:环境必须具备稳定的‘求解-验证不对称性’——模型能编写出一次性的验证器,却无法在自然语言中可靠地执行于新实例。这种不对称性有两种互补形式:部分任务算法上难推理但代码实现简单(如动态规划或图遍历),编译一次即可生成无限校准实例;另一些任务本身难以求解但验证容易,如植入式子集和或约束满足问题。二者均形成求解与验证间的持久鸿沟,使策略无法通过欺骗验证器获得奖励,从而保证奖励信号在模型进化过程中仍具信息量。我们基于此思想构建EvoEnv,一种单策略生成器-求解器方法,从10个种子生成环境,并经过分阶段验证、语义自审、求解器相对难度校准及新颖性检测后才允许使用。最强证据来自已有较强性能的场景:在Qwen3-4B-Thinking上,固定公开数据的RLVR与固定手工环境的RLVR平均表现下降,而EvoEnv反而将表现从72.4提升至74.8,相对提升3.3%。我们认为,真正的自我改进不在于产生更多合成数据,而在于模型学会构建其能力永远无法触及的结构化难题世界。

原文摘要 · Abstract (English)

We pursue a vision for self-improving language models in which the model does not merely generate problems or traces to imitate, but constructs the environments that train it. In zero-data reasoning RL, this reframes self-improvement from a data-generation loop into an environment-construction loop, where each artifact is a reusable executable object that samples instances, computes references, and scores responses. Whether this vision sustains improvement hinges on a single property: the environments must exhibit stable solve--verify asymmetry, the model must be able to write an oracle once that it cannot reliably execute in natural language on fresh instances. This asymmetry takes two complementary forms. Some tasks are algorithmically hard to reason through but trivial as code: a dynamic program or graph traversal, compiled once, yields unboundedly many calibrated instances. Others are intrinsically hard to solve but easy to verify, like planted subset-sum or constraint satisfaction. Both create a durable gap between proposing and solving that the policy cannot close by gaming the verifier, and it is this gap that keeps reward informative as the learner improves. We instantiate this view in EvoEnv, a single-policy generator, solver method that synthesizes Python environments from ten seeds and admits them only after staged validation, semantic self-review, solver-relative difficulty calibration, and novelty checks. The strongest evidence comes from the already-strong regime: on Qwen3-4B-Thinking, fixed public-data RLVR and fixed hand-crafted environment RLVR reduce the average, while EvoEnv improves it from 72.4 to 74.8, a relative gain of 3.3%. Stable self-improvement, we suggest, depends not on producing more synthetic data, but on models learning to construct worlds whose difficulty stays structurally beyond their own reach.

自进化强化学习推理环境构建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。