用统一模型同时建模条件搜索空间中的所有响应面,提升自动调参效率。
Modeling All Response Surfaces in One for Conditional Search Spaces
- 通过结构感知嵌入与自注意力机制,将不同子空间配置映射到统一特征空间。
- 在模拟函数和真实任务上,相比多模型方法提速30%以上,收敛更快。
- 适合处理超参数有依赖关系的自动机器学习场景,如复杂模型调优。
贝叶斯优化(BO)是高效黑箱优化方法,常用于独立超参数搜索空间。但在许多实际自动化机器学习场景中,超参数存在依赖关系,形成条件搜索空间,可划分为结构各异的子空间。各子空间的结构和维度不同,给BO应用带来挑战。已有方法在各子空间分别构建高斯过程(GP)模型,但需大量观测保障性能,且难以捕捉跨子空间的超参数关联。本文提出一种新方法,将所有子空间的响应面统一建模:设计结构感知的超参数嵌入以保留结构信息;引入基于注意力的深度特征提取器,将不同结构的配置投影至统一特征空间,使单个标准高斯过程即可描述响应面。在模拟函数、多个真实任务及HPO-B基准测试中,该方法显著提升了条件搜索空间下BO的有效性与效率。
原文摘要 · Abstract (English)
Bayesian Optimization (BO) is a sample-efficient black-box optimizer commonly used in search spaces where hyperparameters are independent. However, in many practical AutoML scenarios, there will be dependencies among hyperparameters, forming a conditional search space, which can be partitioned into structurally distinct subspaces. The structure and dimensionality of hyperparameter configurations vary across these subspaces, challenging the application of BO. Some previous BO works have proposed solutions to develop multiple Gaussian Process models in these subspaces. However, these approaches tend to be inefficient as they require a substantial number of observations to guarantee each GP's performance and cannot capture relationships between hyperparameters across different subspaces. To address these issues, this paper proposes a novel approach to model the response surfaces of all subspaces in one, which can model the relationships between hyperparameters elegantly via a self-attention mechanism. Concretely, we design a structure-aware hyperparameter embedding to preserve the structural information. Then, we introduce an attention-based deep feature extractor, capable of projecting configurations with different structures from various subspaces into a unified feature space, where the response surfaces can be formulated using a single standard Gaussian Process. The empirical results on a simulation function, various real-world tasks, and HPO-B benchmark demonstrate that our proposed approach improves the efficacy and efficiency of BO within conditional search spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。