用新指标提升代码自进化能力,实现人类级编程表现。
Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine
- 引入CMP指标评估代码代理的自改进潜力,解决性能与潜力不匹配问题。
- 在SWE-bench Verified和Polyglot上超越已有方法,节省计算资源。
- 可迁移至其他数据集和大模型,实现人类级编程水平。
近期研究通过代码代理自我修改代码库实现自进化,其通过偏好更高软件工程基准性能的扩展策略来构建自修改树,假设该性能预示更优的后续自修改。然而,我们发现代理的自进化潜力(元生产力)与其编码基准性能之间存在偏差,即元生产力-性能偏差。受赫胥黎‘类群’概念启发,我们提出一个指标(CMP),通过聚合代理后代的基准性能来衡量其自进化潜力。研究表明,在自进化代码代理开发设置中,真实CMP足以模拟戈德尔机在特定假设下的行为。我们提出赫胥黎-戈德尔机(HGM),通过估计CMP并以此为引导搜索自修改树。在SWE-bench Verified和Polyglot上,HGM优于现有自进化代码代理方法,且使用的CPU小时数更少。最后,HGM展现出强迁移能力,使用GPT-5-mini在SWE-bench Verified上优化的代理,在SWE-bench Lite上以GPT-5评估时达到人类级表现,与人工设计代理的最佳官方结果相当。代码已公开于https://github.com/metauto-ai/HGM。
原文摘要 · Abstract (English)
Recent studies operationalize self-improvement through coding agents that edit their own codebases. They grow a tree of self-modifications through expansion strategies that favor higher software engineering benchmark performance, assuming that this implies more promising subsequent self-modifications. However, we identify a mismatch between the agent's self-improvement potential (metaproductivity) and its coding benchmark performance, namely the Metaproductivity-Performance Mismatch. Inspired by Huxley's concept of clade, we propose a metric ($\mathrm{CMP}$) that aggregates the benchmark performances of the descendants of an agent as an indicator of its potential for self-improvement. We show that, in our self-improving coding agent development setting, access to the true $\mathrm{CMP}$ is sufficient to simulate how the Gödel Machine would behave under certain assumptions. We introduce the Huxley-Gödel Machine (HGM), which, by estimating $\mathrm{CMP}$ and using it as guidance, searches the tree of self-modifications. On SWE-bench Verified and Polyglot, HGM outperforms prior self-improving coding agent development methods while using fewer allocated CPU hours. Last but not least, HGM demonstrates strong transfer to other coding datasets and large language models. The agent optimized by HGM on SWE-bench Verified with GPT-5-mini and evaluated on SWE-bench Lite with GPT-5 achieves human-level performance, matching the best officially checked results of human-engineered coding agents. Our code is publicly available at https://github.com/metauto-ai/HGM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。