缩短时间约束与早停机制提升AutoML基准测试实用性
AutoML Benchmark with shorter time constraints and early stopping
- 引入更短时间预算与早停策略优化AutoML评估
- 11个框架在104任务上表现排名稳定，但早停使性能差异增大
- 适合关注高效建模与资源受限场景的研究者
自动化机器学习（AutoML）可自动在数据上构建机器学习模型。目前评估表格数据AutoML框架的行业标准是AutoML基准（AMLB），其使用1小时和4小时时间预算，在104个任务上进行评估。我们认为应考虑更短时间约束，因其在高频重训练等实际场景中具有价值，并能提升基准的可及性。本文通过缩短时间预算和引入早停机制，减少基准计算量。对11个AutoML框架在104个任务上的评估显示，不同时间预算下框架相对排名基本一致，但使用早停后模型性能差异显著扩大。
原文摘要 · Abstract (English)
Automated Machine Learning (AutoML) automatically builds machine learning (ML) models on data. The de facto standard for evaluating new AutoML frameworks for tabular data is the AutoML Benchmark (AMLB). AMLB proposed to evaluate AutoML frameworks using 1- and 4-hour time budgets across 104 tasks. We argue that shorter time constraints should be considered for the benchmark because of their practical value, such as when models need to be retrained with high frequency, and to make AMLB more accessible. This work considers two ways in which to reduce the overall computation used in the benchmark: smaller time constraints and the use of early stopping. We conduct evaluations of 11 AutoML frameworks on 104 tasks with different time constraints and find the relative ranking of AutoML frameworks is fairly consistent across time constraints, but that using early-stopping leads to a greater variety in model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。