通过重要性分组提升高维超参优化效率
Importance-Aware Scheduling for High-Dimensional Hyperparameter Optimization
- 先小样本估重要性，再分组分配试验资源
- 高维任务上收敛更快，最优解更优
- 适合高维、昂贵评估的超参调优场景
超参数优化（HPO）对构建高性能机器学习/深度学习模型至关重要，但传统优化器在高维空间中因评估成本高、进展分散而表现不佳。本文提出贪心重要性优先（GIF）策略：利用小样本预热估计超参数重要性，按重要性分组，按比例分配试验次数，并保留全空间回退机制。在固定评估预算下，于五个各向异性解析函数、Bayesmark 和 NAS-Bench-301 上评估，GIF 在高维基准上比 TPE、BOHB、随机搜索和顺序分组方法更快收敛并获得更优解。在有效维度较低的 Bayesmark 上仍具竞争力，但优势缩小。消融实验表明重要性估计、比例分配和回退步骤均贡献显著。此外，HIA 组件成功恢复了解析基准中的预期各向异性。结果表明，GIF 是一种简单且可直接集成的高维 HPO 高效优化方法。
原文摘要 · Abstract (English)
Hyperparameter Optimization (HPO) is essential for building high-performing ML/DL models, yet conventional optimizers often struggle in high-dimensional spaces where evaluations are costly and progress is diluted across many low-impact variables. We propose Greedy Importance First (GIF), an importance-aware scheduling strategy that uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally, and retains a full-space fallback. We evaluate GIF under fixed evaluation budgets on five anisotropic analytic functions, Bayesmark, and NAS-Bench-301. On the higher-dimensional benchmarks, GIF reaches better incumbents with faster convergence than TPE, BOHB, Random Search, and Sequential Grouping. On Bayesmark, where the effective dimensionality is smaller, GIF remains competitive but the margins are smaller. Ablation studies show that importance estimation, proportional allocation, and the fallback step all contribute to the gains. We also verify that the HIA component recovers the intended anisotropy on the analytic benchmarks. These results suggest that GIF is a simple and plug-compatible way to improve sample efficiency in high-dimensional HPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。