测试大模型能否自我改进训练算法,发现当前能力仅达最优的四分之一。
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement
- 用10个冻结的算法仓库构建评测基准,让AI改写训练过程
- 平均得分0.166,最佳系统仅达0.250,远未接近理论最优
- 多数模型不改变学习机制,能改的少数平均得分0.226
递归自我改进(RSI)探讨人工智能能否优化生成自身的方法,从而提升后续系统的性能。这一方法的核心是训练算法:更优的目标函数或更新规则能提高每轮计算的产出效率。能否实现RSI取决于智能体是否具备设计训练算法的能力。现有评测体系无法单独检验此能力——它们主要依赖数据收集或超参数调优,且未区分执行方式与学习方式的改变。本文提出AI4AI-Bench,包含10个冻结的研究仓库,覆盖10类训练算法。每个任务中,智能体在单个B300实例上运行4小时,重写训练算法;随后代码从头运行最多12小时,由隐藏的固定评估器评分。10项指标不可通约,每项任务统一映射至0~1尺度:0代表无信息模型,0.1为原仓库算法,1.0为任务最优。在6种系统、29种配置下,平均得分为0.166,最高为0.250,即最强系统仍不足原算法与最优之间的五分之四。分析显示,多数提交未改变学习机制,而改动者平均得分为0.226,未改动者为0.126。更多推理努力显著提升尝试意愿:改动比例从8%升至64%,平均得分从0.094增至0.196。我们公开所有任务、评估器和已评分提交,支持未来重复测量。
原文摘要 · Abstract (English)
Recursive self-improvement (RSI) asks whether an AI system can improve the process that produces AI systems, so that the next system inherits the improvement. That process is the training algorithm: a better objective or update rule improves the compute\mbox{-}capability exchange rate for every subsequent run, including the one that produces the next agent. Whether RSI is feasible therefore turns on whether an agent can design training algorithms. No benchmark isolates that ability: existing suites are won by collecting data or by tuning hyperparameters, and none tells a change to how a run is executed apart from a change to how the model learns. We present AI4AI\mbox{-}Bench, 10 frozen research repositories spanning 10 training algorithm families. In each task, an agent has 4 hours on one B300 to rewrite the training algorithm; its code is then rerun from scratch for up to 12 hours and scored by a fixed evaluator hidden from the agent, against the repository's original algorithm under the same procedure. Because the 10 metrics are incommensurable, every task is mapped onto one scale on which $0$ is an uninformative model, $0.1$ is the algorithm the repository ships, and $1.0$ is the task optimum. Across 29 configurations of 6 systems on all 10 tasks the mean score is $0.166$, and the best system reaches $0.250$: even the strongest closes under a fifth of the distance between the algorithm that was already there and the optimum. The submissions show where that distance went: most never change how the model learns at all, and the minority that do average $0.226$ against $0.126$ for the rest. More reasoning effort mostly buys the willingness to go there, taking that minority from $8\%$ of submissions to $64\%$ and the mean score from $0.094$ to $0.196$. We release the task suite, the evaluators and every scored submission, so that the measurement can be repeated as these systems change.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。