用变分末层训练提升贝叶斯优化，解决复杂相关性问题
Bayesian Optimization via Continual Variational Last Layer Training
- 基于变分贝叶斯末层，连接高斯过程精确推断机制
- 在复杂相关任务上显著超越传统高斯过程和贝叶斯神经网络
- 适合需要高效在线更新的复杂优化场景，如黑箱函数优化
高斯过程（GPs）因能有效建模不确定性且支持在线高效更新，被视为贝叶斯优化（BO）中的先进代理模型。然而，其性能高度依赖核函数选择，对于复杂相关结构往往难以确定或需定制化设计。尽管贝叶斯神经网络（BNNs）具备更高容量潜力，但以往在部分问题类型上表现不佳。本文提出一种新方法，基于变分贝叶斯末层（VBLL），将模型训练与高斯过程的精确条件化相联系，并设计了一种高效的在线训练算法，通过交替执行条件化与优化。实验表明，该方法在具有复杂输入相关性的任务上显著优于标准高斯过程及其他BNN架构，在经典基准任务上性能与精心调参的高斯过程相当。
原文摘要 · Abstract (English)
Gaussian Processes (GPs) are widely seen as the state-of-the-art surrogate models for Bayesian optimization (BO) due to their ability to model uncertainty and their performance on tasks where correlations are easily captured (such as those defined by Euclidean metrics) and their ability to be efficiently updated online. However, the performance of GPs depends on the choice of kernel, and kernel selection for complex correlation structures is often difficult or must be made bespoke. While Bayesian neural networks (BNNs) are a promising direction for higher capacity surrogate models, they have so far seen limited use due to poor performance on some problem types. In this paper, we propose an approach which shows competitive performance on many problem types, including some that BNNs typically struggle with. We build on variational Bayesian last layers (VBLLs), and connect training of these models to exact conditioning in GPs. We exploit this connection to develop an efficient online training algorithm that interleaves conditioning and optimization. Our findings suggest that VBLL networks significantly outperform GPs and other BNN architectures on tasks with complex input correlations, and match the performance of well-tuned GPs on established benchmark tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。