评测大模型代理在调参中的持续优化能力
AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
- 设计30个可执行的机器学习任务，评估代理逐轮调参决策
- 12个代理均能改进性能，但长期迭代能力有限
- 适合研究大模型实验自主性与自动调参的学者
随着大语言模型从代码补全系统演变为自主科学代理，评估其开展实验的能力变得愈发重要。现有基准多聚焦静态代码生成、论文复现或最终答案正确性，未能直接检验代理是否能解读实验证据并指导后续超参数调整。为此，我们提出AgentHPOBench，一个包含30个可执行机器学习任务的序列化基准，覆盖七个研究领域。每个任务始于一个验证过的基线运行，随后代理执行多次序列干预。每一步中，代理观察累积的配置、指标和日志后提出下一个有效配置。我们在统一协议下评估了12个常用代理和传统超参数优化基线。结果表明，当前代理在各领域展现出可测量的实验优化能力，但在持续迭代优化、复杂日志诊断和稳定逼近参考性能方面仍存在明显局限。
原文摘要 · Abstract (English)
As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。