对比多种AI科研代理策略,发现搜索密度决定哪种方法更有效。
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics

- 设计分离策略与执行环境的基准,专注分析搜索行为
- 简单贪心搜索在密集优化机会下表现接近最优树搜索
- 首次揭示早期收敛和方向聚焦探索是性能关键,适合研究智能体行为
AI科研代理通过自动化假设生成、实验与实证优化加速机器学习研究。现有代理策略从贪婪爬山到树搜索和进化优化不等,但何种策略驱动性能尚不明确。回答此问题需一个基准,将代理策略(如搜索拓扑)与执行基础设施(如代码编辑器)分离,使性能差异归因于策略而非环境,并提供过程级指标以分析探索行为。现有基准支持有限。本文提出FML-Bench,涵盖10个领域中的18个基础ML研究任务,分离策略与基础设施,并定义12项过程级行为指标。评估六种代表性代理发现:(1) 策略复杂度本身不保证强性能:简单贪心爬山几乎匹配表现最佳的树搜索代理,均显著优于其余代理;(2) 分析表明该现象与改进机会结构相关:当机会密集时贪心搜索更优,稀疏时树搜索与进化策略更优;基于此构建的自适应代理在检测到改进停滞后切换至广度探索,表现优于其余六种代理,初步支持该观察;(3) 过程级分析显示,早期收敛和方向聚焦探索显著关联最终性能,而解多样性与计算成本则无关。基准已开源:https://github.com/qrzou/FML-bench。
原文摘要 · Abstract (English)
AI research agents accelerate ML research by automating hypothesis generation, experimentation, and empirical refinement. Existing agent strategies range from greedy hill-climbing to tree search and evolutionary optimization, yet which strategy choices drive performance remains unclear. Answering this question requires a benchmark that separates agent strategy (e.g., search topology) from execution infrastructure (e.g., code editor), so that performance differences are attributable to strategy rather than infrastructure, and that provides process-level metrics beyond final scores to analyze exploration behaviors. Existing benchmarks offer limited support. We propose FML-Bench, a benchmark of 18 fundamental ML research tasks across 10 domains that separates agent strategy from execution infrastructure and defines 12 process-level behavioral metrics. Evaluating six representative agents, we find that: (1) strategy complexity alone does not guarantee strong performance: a simple greedy hill-climber nearly matches the best-performing tree-search agent, both well above the remaining agents; (2) our analysis suggests this pattern relates to improvement opportunity structure: greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse; an adaptive agent built on this insight switches to broader exploration upon detecting improvement stagnation and outperforms the other six agents, lending initial support to this observation; and (3) process-level analysis reveals that early convergence and directionally focused exploration are significantly associated with final performance, while solution diversity and compute cost are not. Our benchmark is available at: https://github.com/qrzou/FML-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。