AI设计的AI方法多数是人类方法的重组,偶有突破但难推广。
When AI Designs AI: Innovation or Imitation?
- 通过构建人类方法的设计空间,量化比较人与AI设计的算法差异。
- 仅10/72配置达到或超越人类最佳性能,且不具跨任务泛化能力。
- 96.8%的AI设计方法在人类设计空间内,近半数完全复现已有方案。
近年来,大型语言模型代理在复杂AI任务的方法设计上愈发成熟,引发两个核心问题:代理设计的方法表现如何,其算法设计是否与人类设计不同。本文提出一种分析框架,从人类设计方法中推导出任务特异的算法设计空间,将人与代理设计的方法映射到该空间,并在模块层面量化其算法差异。对多个代表性、开放式多模态任务上的广泛评估显示,当前代理在10/72配置中可达到或超越人类最先进(SOTA)性能,但该成功无法可靠泛化至其他任务或代理。此外,96.8%的代理设计方法位于人类衍生的设计空间内,主要重新组合人类已有的算法选择,近一半完全匹配已有算法设计。结果表明,尽管代理偶尔能超越人类性能,其算法设计仍局限于人类设计空间,反映的是对已有算法的选择性重用与重组。
原文摘要 · Abstract (English)
Recent advances in LLM agents have made them increasingly capable of designing methods for complex AI tasks. This raises two central questions about agent-designed methods relative to human-designed methods: how well they perform, and how different their algorithmic designs are. To study these questions, this paper introduces an analysis that derives task-specific algorithmic design spaces from human-designed methods, maps both human- and agent-designed methods into these spaces, and quantifies their algorithmic differences at the module level. Widely used LLM agents are evaluated on a suite of representative, open-ended AI tasks spanning multiple modalities, and the methods they design are analyzed in terms of both task performance and algorithmic differences from human-designed methods. Experimental results show that current agents can occasionally match or surpass human state-of-the-art (SOTA) performance (10/72 configurations), but such success does not generalize reliably across tasks or agents. Moreover, 96.8% of agent-designed methods fall within human-derived algorithmic design spaces, largely recombining algorithmic choices found in human-designed methods, while nearly half exactly match an existing human algorithmic design. Taken together, these findings suggest that although current agents can occasionally match or surpass human SOTA performance, their algorithmic designs remain within human-derived algorithmic design spaces, reflecting the reuse and recombination of algorithmic choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。