通过重要性评估加速深度学习超参优化,节省三成时间。
Grouped Sequential Optimization Strategy -- the Application of Hyperparameter Importance Assessment in Deep Learning
- 按超参重要性分组依次优化,减少无效搜索。
- 在6个数据集上平均提速31.9%,性能不变。
- 适合追求高效训练的算法工程师和研究者。
超参数优化(HPO)是机器学习流程中的关键环节,显著影响模型的鲁棒性、稳定性和泛化能力。然而,传统方法如网格搜索和随机搜索效率低下,贝叶斯优化在高维空间中仍受限。本文基于先前对10个超参在CNN上使用10个常见图像分类数据集的量化重要性分析,提出一种名为‘分组顺序优化’的新策略。该策略利用超参重要性评估(HIA)结果指导优化顺序,先优化影响大的超参,再逐步处理次要项。在6个额外图像分类数据集上的实验表明,引入HIA可显著加速HPO,平均减少31.9%的优化时间,且不牺牲模型性能。
原文摘要 · Abstract (English)
Hyperparameter optimization (HPO) is a critical component of machine learning pipelines, significantly affecting model robustness, stability, and generalization. However, HPO is often a time-consuming and computationally intensive task. Traditional HPO methods, such as grid search and random search, often suffer from inefficiency. Bayesian optimization, while more efficient, still struggles with high-dimensional search spaces. In this paper, we contribute to the field by exploring how insights gained from hyperparameter importance assessment (HIA) can be leveraged to accelerate HPO, reducing both time and computational resources. Building on prior work that quantified hyperparameter importance by evaluating 10 hyperparameters on CNNs using 10 common image classification datasets, we implement a novel HPO strategy called 'Sequential Grouping.' That prior work assessed the importance weights of the investigated hyperparameters based on their influence on model performance, providing valuable insights that we leverage to optimize our HPO process. Our experiments, validated across six additional image classification datasets, demonstrate that incorporating hyperparameter importance assessment (HIA) can significantly accelerate HPO without compromising model performance, reducing optimization time by an average of 31.9\% compared to the conventional simultaneous strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。