提出新初始化方法,提升贝叶斯优化在少量样本下的预测与搜索效率。
Informed Initialization for Bayesian Optimization and Active Learning
- 基于信息论设计新策略,兼顾不确定性降低与超参学习。
- 在小批量场景下,预测精度和优化性能显著优于传统随机初始化。
- 适合真实世界中评估成本高、样本少的贝叶斯优化任务。
贝叶斯优化广泛用于优化昂贵的黑箱函数,依赖高斯过程等概率代理模型。在仅能评估少量批次点的少样本场景下,初始化质量对代理模型预测性能至关重要。然而,实践者通常采用(准)随机设计覆盖输入空间,这忽略了两个关键问题:(a) 均匀填充未必能最小化预测不确定度;(b) 高效的超参数学习对高质量预测至关重要,但可能与空间填充设计冲突。为此,我们提出超参感知预测探索(HIPE),一种利用信息论原则平衡预测不确定度减少与超参数学习的新采集策略。我们在高斯过程设置下推导出HIPE的闭式表达,并通过在主动学习和少样本贝叶斯优化中的大量实验验证其有效性。结果表明,相较于标准初始化策略,HIPE在预测准确性、超参数识别及后续优化性能方面均有显著提升,尤其在大批次、少样本设置下表现突出,适用于众多现实世界的贝叶斯优化应用。
原文摘要 · Abstract (English)
Bayesian Optimization is a widely used method for optimizing expensive black-box functions, relying on probabilistic surrogate models such as Gaussian Processes. The quality of the surrogate model is crucial for good optimization performance, especially in the few-shot setting where only a small number of batches of points can be evaluated. In this setting, the initialization plays a critical role in shaping the surrogate's predictive quality and guiding subsequent optimization. Despite this, practitioners typically rely on (quasi-)random designs to cover the input space. However, such approaches neglect two key factors: (a) space-filling designs may not be desirable to reduce predictive uncertainty, and (b) efficient hyperparameter learning during initialization is essential for high-quality prediction, which may conflict with space-filling designs. To address these limitations, we propose Hyperparameter-Informed Predictive Exploration (HIPE), a novel acquisition strategy that balances predictive uncertainty reduction with hyperparameter learning using information-theoretic principles. We derive a closed-form expression for HIPE in the Gaussian Process setting and demonstrate its effectiveness through extensive experiments in active learning and few-shot BO. Our results show that HIPE outperforms standard initialization strategies in terms of predictive accuracy, hyperparameter identification, and subsequent optimization performance, particularly in large-batch, few-shot settings relevant to many real-world Bayesian Optimization applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。