arXiv:2505.23673cs.LG2025-05ICML被引 4

用人类偏好反馈实现贝叶斯优化,理论证明样本效率接近最优。

Bayesian Optimization from Human Feedback: Near-Optimal Regret Bounds

  • 基于人类偏好构建贝叶斯优化框架,仅需比较两个动作优劣
  • 首次得到$ ilde{ m O}( ext{sqrt}(Γ(T)T))$的近似最优后悔界
  • 常见核函数下,偏好样本效率可媲美完整数值反馈

基于人类反馈的贝叶斯优化(BOHF)近年来受到广泛关注,其核心是通过有限次偏好查询(每次仅提供两个动作的相对优劣)来寻找最优策略。本文在经典布拉德利-特雷西-卢斯(BTL)反馈模型下,推导出新的后悔上界:$ ilde{ m O}( ext{sqrt}(Γ(T)T))$,其中$Γ(T)$为最大信息增益(与核函数相关的复杂度项),$T$为总查询次数。该结果显著优于现有方法。尤其对于常见核函数,我们证明了偏好反馈下的样本复杂度可达到与传统标量反馈相同的最优阶数,即使用相同数量的偏好样本即可获得近乎最优解。

原文摘要 · Abstract (English)

Bayesian optimization (BO) with preference-based feedback has recently garnered significant attention due to its emerging applications. We refer to this problem as Bayesian Optimization from Human Feedback (BOHF), which differs from conventional BO by learning the best actions from a reduced feedback model, where only the preference between two actions is revealed to the learner at each time step. The objective is to identify the best action using a limited number of preference queries, typically obtained through costly human feedback. Existing work, which adopts the Bradley-Terry-Luce (BTL) feedback model, provides regret bounds for the performance of several algorithms. In this work, within the same framework we develop tighter performance guarantees. Specifically, we derive regret bounds of $\tilde{\mathcal{O}}(\sqrt{Γ(T)T})$, where $Γ(T)$ represents the maximum information gain$\unicode{x2014}$a kernel-specific complexity term$\unicode{x2014}$and $T$ is the number of queries. Our results significantly improve upon existing bounds. Notably, for common kernels, we show that the order-optimal sample complexities of conventional BO$\unicode{x2014}$achieved with richer feedback models$\unicode{x2014}$are recovered. In other words, the same number of preferential samples as scalar-valued samples is sufficient to find a nearly optimal solution.

贝叶斯优化人类反馈理论分析偏好学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。