用理论证明:好先验能大幅减少超参优化的计算量
Provably Reduced Sample Cost in Prior-Guided Hyperparameter Optimization

- 基于先验信息构建性能分布模型,量化其对评估次数的影响
- 实验显示在真实数据集上可节省高达90%的计算预算
- 适合关注绿色自动化机器学习与高效超参优化的研究者
大规模超参数优化在自动化机器学习中消耗大量计算资源,引发对可扩展性和能效的担忧。现有方法虽利用先验信息加速黑箱和多保真度设置下的优化,但缺乏对先验信息如何定量降低样本复杂度的刻画。本文首次通过固定预算最优臂识别的理论框架,为带先验的多保真度超参优化提供了分布依赖的样本复杂度界。我们直接将先验建模为配置性能的均值分布,推导出显式的、分布依赖的误差界,量化了先验与评估预算之间的关系。分析表明,集中概率质量于近优解的有信息先验可显著减少所需评估次数;而无信息或误导性先验则退化为基线表现。我们在合成基准和LCBench(深度学习常见多保真度超参优化基准)上进行概念验证实验,证实理论结果,在保持解质量的同时实现最高达90%的预算缩减。整体成果为先验引导且计算高效的绿色自动化机器学习提供了原理性基础。
原文摘要 · Abstract (English)
Large-scale hyperparameter optimization (HPO) in automated machine learning (AutoML) consumes substantial computational resources, raising growing concerns about scalability and energy efficiency. Existing methods use prior information heuristically to accelerate both black-box and multi-fidelity settings, but they lack a characterization of how prior informativeness quantitatively reduces sample complexity. In this work, we provide the first distribution-dependent sample complexity bounds for multi-fidelity HPO with priors through the formal lens of fixed-budget best-arm identification. By modeling priors directly over arm means as configuration performance, we derive explicit, distribution-dependent error bounds that quantify the relationship between priors and evaluation budget. Our analysis shows that informative priors, which concentrate probability mass on near-optimal arms, yield reductions in the number of required evaluations, whereas baseline performance is recovered with uninformative or misleading priors. We conduct proof-of-concept experiments on a synthetic benchmark and on LCBench, a common multi-fidelity HPO benchmark for deep learning, to confirm our theoretical results, achieving up to 90% budget reduction while retaining solution quality. Together, our results provide a principled foundation for prior-guided and compute-efficient green AutoML.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。