arXiv:2512.18552cs.SEcs.AI2025-12中稿 · ICML被引 34

让代码智能体通过自我对弈自主学习修bug,摆脱人工标注依赖。

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

  • 用自博弈RL训练单个智能体在真实代码库中迭代制造和修复复杂漏洞。
  • 在SWE-bench Verified和Pro上分别提升10.4和7.8分,全程超越人类数据基线。
  • 无需人工标注问题或测试,适合研究自主智能软件系统构建者。

当前由大语言模型和代理强化学习驱动的软件智能体虽能提升开发效率,但其训练数据(如GitHub议题与提交记录)和环境(如通过/失败测试)高度依赖人工知识或整理,成为通向超级智能的根本障碍。本文提出自博弈软件强化学习(SSR),是迈向超智能软件代理训练范式的第一步。该方法仅需访问沙盒化代码仓库及其依赖项,无需人工标注的议题或测试。基于真实代码库,一个大语言模型代理在自博弈环境中通过强化学习,持续生成并修复日益复杂的软件漏洞,每个漏洞均由测试补丁形式正式定义,而非自然语言描述。在SWE-bench Verified与SWE-Bench Pro基准上,SSR分别实现+10.4和+7.8的显著自我提升,且在整个训练过程中始终优于依赖人类数据的基线,尽管评估时使用的是自博弈阶段未接触过的自然语言议题。结果表明,智能体可从真实世界代码库中自主获取海量学习经验,最终实现超越人类的理解能力、解决新挑战及从零创造新软件的超智能系统。

原文摘要 · Abstract (English)

While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.g., GitHub issues and pull requests) and environments (e.g., pass-to-pass and fail-to-pass tests) heavily depend on human knowledge or curation, posing a fundamental barrier to superintelligence. In this paper, we present Self-play SWE-RL (SSR), a first step toward training paradigms for superintelligent software agents. Our approach takes minimal data assumptions, only requiring access to sandboxed repositories with source code and installed dependencies, with no need for human-labeled issues or tests. Grounded in these real-world codebases, a single LLM agent is trained via reinforcement learning in a self-play setting to iteratively inject and repair software bugs of increasing complexity, with each bug formally specified by a test patch rather than a natural language issue description. On the SWE-bench Verified and SWE-Bench Pro benchmarks, SSR achieves notable self-improvement (+10.4 and +7.8 points, respectively) and consistently outperforms the human-data baseline over the entire training trajectory, despite being evaluated on natural language issues absent from self-play. Our results, albeit early, suggest a path where agents autonomously gather extensive learning experiences from real-world software repositories, ultimately enabling superintelligent systems that exceed human capabilities in understanding how systems are constructed, solving novel challenges, and autonomously creating new software from scratch.

智能编程自博弈强化学习代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。