评估自主研究系统时,既要看结果质量,也要看搜索效率。
Efficiency Matters in Autonomous Research
- 用帕累托曲线下的面积衡量搜索过程的效率
- 不同搜索算法在各任务中表现各异,无最优统一方案
- 提出动态调整策略,适合预算有限的真实科研场景
AI驱动的自主研究(AR)系统在多种任务中日益有效,但其性能通常仅以最终结果质量来评估。本文指出,解空间搜索过程的效率同样重要却常被忽视。一个优秀的AR系统不仅需产出高质量结果,更应在最小化评估预算的前提下达成目标。随着AR从数学、编程等低成本验证领域拓展至需昂贵物理实验的现实科学场景,搜索效率愈发关键。为此,本文提出结合帕累托前沿曲线下面积(AUC)与最终结果质量进行综合评估。在十二个系统优化任务中,对比了爬山法、束搜索、树搜索和进化搜索等算法,发现并无单一搜索结构始终最优。同时表明,搜索效率与最终结果质量是独立维度:某些方法虽最终表现最佳,但前期消耗大量评估资源。由于最佳搜索策略难以预知,本文引入名为「流式搜索」的自适应机制,通过组合多搜索路径并利用谱带模型动态分配固定评估预算。实验显示,流式搜索在多数任务中实现最高整体效率,接近为每项任务预先选定最优结构的‘理想基准’表现。
原文摘要 · Abstract (English)
AI-driven autonomous research (AR) systems are becoming increasingly effective across a broad range of tasks. Their performance, however, is still evaluated primarily by the quality of the final outcome. In this paper, we argue that the efficiency of the solution-search process is an equally important but often overlooked dimension of performance. A strong AR system should not only produce high-quality results, but also reach them with as small a budget as possible. Search efficiency will become increasingly important as AR expands from domains with inexpensive verification, such as mathematics and coding, to real-world scientific settings in which solution evaluation may require costly physical experiments. To capture this dimension, we propose evaluating AR systems using the area under the curve (AUC) of the Pareto frontier, alongside final outcome quality. We compare several families of search algorithms, including hill climbing, beam search, tree search, and evolutionary search, across twelve systems-optimization tasks. We find that no single search structure is consistently the most efficient. We also show that search efficiency and final outcome quality are distinct performance dimensions: a method that eventually achieves the best result may nevertheless improve slowly and consume substantially more evaluation budget before reaching that result. Because the most effective search policy is generally unknown in advance, we introduce an adaptive procedure called fluid search, which uses a portfolio bandit to dynamically allocate a fixed evaluation budget across a forest of search processes. Across the evaluated tasks, fluid search achieves the highest overall search efficiency, closely matching the performance of a per-task oracle that is given the best search structure for each task in advance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。