让编程智能体在运行时自我进化,实时提升解决实际问题的能力。
Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
- 智能体在运行中自主优化自身结构,无需预先设计
- 在SWE-bench Verified上达到77.4%求解率,超越所有现有模型
- 适合追求自适应能力的软件自动化研究者与开发者
大型语言模型正在重塑几乎所有行业,包括软件工程。近年来,许多基于LLM的智能体被提出以解决真实世界的软件问题。这些编程智能体通常配备一套编码工具,可自主决策下一步动作,形成完整的任务执行轨迹。尽管前景广阔,它们通常需要专门设计,且可能表现不佳,因为探索整个智能体架构设计空间极其困难且成本高昂。鉴于智能体本身也是软件,可进一步优化,近期已有研究提出自改进型智能体,如达尔文-哥德尔机(DGM)。然而,这类智能体需在特定基准上进行昂贵的离线训练,且泛化能力有限。本文提出Live-SWE-agent,首个能在运行时实时、持续自我进化的动态编程智能体。它从仅具备bash工具访问权限的最简架构(mini-SWE-agent)出发,在解决真实软件问题过程中自主演化其架构实现。在广泛使用的SWE-bench Verified基准上的评估显示,Live-SWE-agent在不使用测试时缩放的情况下达到了77.4%的求解率,优于所有现有智能体,包括最佳专有方案。此外,它在最新的SWE-Bench Pro基准上也超越了最先进的手动设计智能体,达到45.8%的最佳已知求解率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are reshaping almost all industries, including software engineering. In recent years, a number of LLM agents have been proposed to solve real-world software problems. Such software agents are typically equipped with a suite of coding tools and can autonomously decide the next actions to form complete trajectories to solve end-to-end software tasks. While promising, they typically require dedicated design and may still be suboptimal, since it can be extremely challenging and costly to exhaust the entire agent scaffold design space. Recognizing that software agents are inherently software themselves that can be further refined/modified, researchers have proposed a number of self-improving software agents recently, including the Darwin-Gödel Machine (DGM). Meanwhile, such self-improving agents require costly offline training on specific benchmarks and may not generalize well across different LLMs or benchmarks. In this paper, we propose Live-SWE-agent, the first live software agent that can autonomously and continuously evolve itself on-the-fly during runtime when solving real-world software problems. More specifically, Live-SWE-agent starts with the most basic agent scaffold with only access to bash tools (e.g., mini-SWE-agent), and autonomously evolves its own scaffold implementation while solving real-world software problems. Our evaluation on the widely studied SWE-bench Verified benchmark shows that LIVE-SWE-AGENT can achieve an impressive solve rate of 77.4% without test-time scaling, outperforming all existing software agents, including the best proprietary solution. Moreover, Live-SWE-agent outperforms state-of-the-art manually crafted software agents on the recent SWE-Bench Pro benchmark, achieving the best-known solve rate of 45.8%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。