arXiv:2411.16586stat.MLcs.LG2024-11被引 3

用α散度改进贝叶斯优化,更智能地选下一个测试点。

Alpha Entropy Search for New Information-based Bayesian Optimization

  • 基于α散度构建新采集函数,替代传统KL散度。
  • 在合成、基准和真实场景中表现优于或媲美现有方法。
  • 支持灵活调节参数,适合超参调优等复杂优化任务。

基于信息论的贝叶斯优化(BO)方法在多个任务中已达到顶尖性能,其核心依赖于Kullback-Leibler(KL)散度计算采集函数。本文提出一种新型信息基采集函数——Alpha熵搜索(AES),基于α散度,该散度泛化了KL散度。AES通过α散度衡量目标值与全局最优位置及对应值之间的依赖性,选择对全局最优最具有信息量的评估点。α参数控制散度行为,在单峰差异与全局差异间权衡,不同α值生成不同采集函数。尽管AES无闭式表达,我们提出使用截断高斯分布进行高效准确近似。实践中可由用户选择α值,本文建议同时考虑多个α值组合使用。我们在BOTorch中实现AES,通过合成数据、基准测试及深度神经网络超参调优等真实场景验证其性能。结果表明,AES表现与JES、MES、PES等先进方法相当甚至更优。

原文摘要 · Abstract (English)

Bayesian optimization (BO) methods based on information theory have obtained state-of-the-art results in several tasks. These techniques heavily rely on the Kullback-Leibler (KL) divergence to compute the acquisition function. In this work, we introduce a novel information-based class of acquisition functions for BO called Alpha Entropy Search (AES). AES is based on the α-divergence, that generalizes the KL divergence. Iteratively, AES selects the next evaluation point as the one whose associated target value has the highest level of the dependency with respect to the location and associated value of the global maximum of the optimization problem. Dependency is measured in terms of the α-divergence, as an alternative to the KL divergence. Intuitively, this favors the evaluation of the objective function at the most informative points about the global maximum. The α-divergence has a free parameter α, which determines the behavior of the divergence, trading-off evaluating differences between distributions at a single mode, and evaluating differences globally. Therefore, different values of α result in different acquisition functions. AES acquisition lacks a closed-form expression. However, we propose an efficient and accurate approximation using a truncated Gaussian distribution. In practice, the value of α can be chosen by the practitioner, but here we suggest to use a combination of acquisition functions obtained by simultaneously considering a range of values of α. We provide an implementation of AES in BOTorch and we evaluate its performance in both synthetic, benchmark and real-world experiments involving the tuning of the hyper-parameters of a deep neural network. These experiments show that the performance of AES is competitive with respect to other information-based acquisition functions such as JES, MES or PES.

贝叶斯优化信息论超参调优α散度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。