arXiv:2506.11442cs.SEcs.LG2025-06被引 13

让AI代码生成能自我验证并持续优化,突破推理边界。

ReVeal: Self-Evolving Code Agents via Reliable Self-Verification

  • 通过多轮自验证机制,让AI代码生成与测试协同进化。
  • 在LiveCodeBench上实现20+轮自主演化,仅用3轮训练数据。
  • 适合构建可自主进化的智能编程代理,推动高阶推理发展。

基于可验证奖励的强化学习(RLVR)提升了大语言模型的推理能力。然而,现有方法仅依赖结果奖励,未显式优化自验证,也未能利用真实环境中的可靠信号,导致自验证不可靠,测试时扩展能力有限。为此,本文通过扩大生成与验证的不对称性,显式优化自验证,使其成为深层测试时扩展的可靠驱动力。提出ReVeal框架,采用多轮强化学习机制,通过自验证与工具评估迭代演化代码生成。该框架将长时序推理建模为生成-验证的交替过程,并引入TAPO实现回合级信用分配,促进代码与测试生成的共演化。推理阶段,强化后的自验证使模型能利用自构建测试和工具反馈,在LiveCodeBench上持续演化代码超过20轮,尽管训练仅使用3轮数据。同时显著提升Pass@k指标,表明更强探索能力拓展了基础模型的推理边界。这些发现证明ReVeal是一种可扩展的强化学习训练与测试时扩展范式,为更鲁棒、自主的AI代理铺平道路。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models. However, existing methods rely solely on outcome rewards, without explicitly optimizing verification or leveraging reliable signals from realistic environments, leading to unreliable self-verification and limited test-time scaling. To address this, we widen the verification-generation asymmetry by explicitly optimizing self-verification, making it a reliable driver of deeper test-time scaling. We introduce ReVeal, a multi-turn reinforcement learning framework that evolves code generation through self-verification and tool-based evaluation. ReVeal structures long-horizon reasoning as iterative generation-verification turns and incorporates TAPO for turn-level credit assignment, fostering the co-evolution of code and test generation. At inference, this strengthened self-verification enables the model to use self-constructed tests and tool feedback to continuously evolve code for 20+ turns on LiveCodeBench despite training on only three. It also significantly improves Pass@k, indicating stronger exploration that expands the reasoning boundaries of the base model. These findings highlight the promise of ReVeal as a scalable paradigm for RL training and test-time scaling, paving the way for more robust and autonomous AI agents.

代码生成强化学习自验证推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。