改进高斯过程强化学习的后悔上界，实现无噪声与非平稳噪声下的最优性能。
Improved Regret Analysis in Gaussian Process Bandits: Optimality for Noiseless Reward, RKHS norm, and Non-Stationary Variance
- 提出新的后验方差上界，降低对噪声方差的依赖
- 在无噪声和非平稳噪声下达到最优后悔上界
- 适用于奖励函数在RKHS空间且噪声变化的场景
我们研究高斯过程（GP）强化学习问题，目标是在未知奖励函数属于某再生核希尔伯特空间（RKHS）的情况下最小化后悔。最大后验方差分析对近似最优的GP强化学习算法（如最大方差减少法MVR和分阶段消除法PE）至关重要。本文首次给出最大后验方差的新上界，显著改善了对噪声方差参数的依赖性。基于该结果，我们优化了MVR和PE算法，实现了：(i) 无噪声设定下的近乎最优后悔上界；(ii) 与奖励函数的RKHS范数相关的最优后悔上界。此外，作为该上界的另一应用，我们分析了时变噪声方差下的GP强化学习问题，这是线性强化学习中异方差噪声的核化扩展。在此设定下，我们证明了基于MVR和PE的算法可获得依赖噪声方差的后悔上界，且该上界与我们的后悔下界一致。
原文摘要 · Abstract (English)
We study the Gaussian process (GP) bandit problem, whose goal is to minimize regret under an unknown reward function lying in some reproducing kernel Hilbert space (RKHS). The maximum posterior variance analysis is vital in analyzing near-optimal GP bandit algorithms such as maximum variance reduction (MVR) and phased elimination (PE). Therefore, we first show the new upper bound of the maximum posterior variance, which improves the dependence of the noise variance parameters of the GP. By leveraging this result, we refine the MVR and PE to obtain (i) a nearly optimal regret upper bound in the noiseless setting and (ii) regret upper bounds that are optimal with respect to the RKHS norm of the reward function. Furthermore, as another application of our proposed bound, we analyze the GP bandit under the time-varying noise variance setting, which is the kernelized extension of the linear bandit with heteroscedastic noise. For this problem, we show that MVR and PE-based algorithms achieve noise variance-dependent regret upper bounds, which matches our regret lower bound.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。