arXiv:2603.15916cs.LGcs.AI2026-03被引 1

LLM自主设计实验时,真正是在搜模型架构而非调参数。

Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experiments

  • 用统计分解法分析万次实验,发现模型结构差异贡献94%性能变化
  • 在27天内通过自研搜索找到新配置,使检测准确率提升至0.985
  • 首次建立大尺度框架,揭示多智能体搜索的动态规律

当大语言模型代理自主设计机器学习实验时,它们究竟在进行真正的架构搜索,还是仅在狭窄区域内调参?我们通过分析两个大模型(Claude Opus 和 Gemini 2.5 Pro)在27天内执行的10,469次实验,回答了这一问题。实验覆盖108,000个离散组合配置,用于行车记录仪碰撞检测。基于ANOVA分解,发现架构选择解释了94%的性能方差(F = 1324,η² = 0.94),而固定架构下的超参数变化仅解释6%。在第二个碰撞数据集上交叉验证,架构解释率仍达75%,且胜出骨干网络不同,证明为真实的架构发现。代理关键成果是发现V-JEPA2视频特征搭配Zipformer时间编码器可实现0.9245 AP——这是人类从未提出的配置。在N=50时,基于LLM的搜索达到0.985 AP,优于从零开始的随机搜索(0.965)。修复漏洞后收敛符合幂律(c = 0.11,R² = 0.93),低指数反映广域探索成本高,而非效率低下。与随机或贝叶斯基线相比,该方法发现更优的架构区域。通过熵周期与詹森-香农专业化刻画多智能体搜索动态,首次提供大规模实证框架支持LLM引导的组合式机器学习实验设计。

原文摘要 · Abstract (English)

When LLM agents autonomously design ML experiments, do they perform genuine architecture search -- or do they default to hyperparameter tuning within a narrow region of the design space? We answer this question by analyzing 10,469 experiments executed by two LLM agents (Claude Opus and Gemini 2.5 Pro) across a combinatorial configuration space of 108,000 discrete cells for dashcam collision detection over 27 days. Through ANOVA decomposition, we find that \textbf{architectural choices explain 94\% of performance variance} ($F = 1324$, $η^2 = 0.94$), while hyperparameter variation within a fixed architecture explains only 6\%. Cross-task validation on a second collision dataset confirms this finding (75\% architecture-explained variance) with a \emph{different} winning backbone, confirming genuine architecture discovery. The agents' key contribution is discovering that V-JEPA\,2 video features with Zipformer temporal encoders achieve 0.9245 AP -- a configuration no human proposed -- and concentrating search on productive architectural regions: at $N = 50$, LLM-guided search reaches AP $= 0.985$ versus $0.965$ for from-scratch random search. Post-bugfix convergence follows a power law ($c = 0.11$, $R^2 = 0.93$); the low exponent reflects the cost of broad exploration, not inefficiency, since the LLM discovers qualitatively better regions than random or Bayesian baselines. We characterize multi-agent search dynamics via entropy cycles and Jensen--Shannon specialization, providing the first large-scale empirical framework for LLM-guided combinatorial ML experiment design.

大模型实验架构搜索自动机器学习行为分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。