arXiv:2502.10020stat.MLcs.LG2025-02ICML被引 16

改进了多项式逻辑模型的在线置信区间,实现更优的强化学习后悔界。

Improved Online Confidence Bounds for Multinomial Logistic Bandits

  • 基于自协方差性质与Ville不等式,构建更紧的置信边界。
  • 提出常数时间算法OFU-MNL++,实现方差依赖的最优后悔上界。
  • 适合关注高维上下文强化学习与在线决策的研究者。

本文提出了针对多项式逻辑(MNL)模型的改进在线置信边界,并将其应用于MNL贝叶斯问题,实现了方差依赖的最优后悔上界。近期,Lee & Oh(2024)建立了MNL模型的在线置信边界,在MNL贝叶斯中实现了近最小最大最优后悔。然而,其结果仍依赖于未知参数的范数有界性 $B$ 和可能结果的最大数量 $K$。为解决此问题,我们首先推导出 $Oig( oot{d} ext{log } t + B oot{d}ig)$ 的在线置信边界,显著优于此前 $O(B oot{d} ext{log } t ext{ log } K)$(Lee & Oh, 2024)的结果。这主要得益于对MNL损失函数更紧的自协方差性质分析,并应用Ville不等式控制估计误差。利用该新置信边界,我们提出常数时间算法OFU-MNL++,在足够大的 $T$ 时,达到方差依赖的后悔上界 $Oig( d ext{log } T oot{ extstyle extsum}_{t=1}^T σ_t^2 }ig)$,其中 $σ_t^2$ 表示第 $t$ 轮奖励的方差,$d$ 为上下文维度,$T$ 为总轮数。此外,我们引入基于最大似然估计的算法OFU-MN$^2$L,实现了任意时刻无 $B$ 的多项式依赖后悔上界 $Oig( d ext{log}(BT) oot{ extstyle extsum}_{t=1}^T σ_t^2 }ig)$。

原文摘要 · Abstract (English)

In this paper, we propose an improved online confidence bound for multinomial logistic (MNL) models and apply this result to MNL bandits, achieving variance-dependent optimal regret. Recently, Lee & Oh (2024) established an online confidence bound for MNL models and achieved nearly minimax-optimal regret in MNL bandits. However, their results still depend on the norm-boundedness of the unknown parameter $B$ and the maximum size of possible outcomes $K$. To address this, we first derive an online confidence bound of $O\left(\sqrt{d \log t} + B \sqrt{d} \right)$, which is a significant improvement over the previous bound of $O (B \sqrt{d} \log t \log K )$ (Lee & Oh, 2024). This is mainly achieved by establishing tighter self-concordant properties of the MNL loss and applying Ville's inequality to bound the estimation error. Using this new online confidence bound, we propose a constant-time algorithm, OFU-MNL++, which achieves a variance-dependent regret bound of $O \Big( d \log T \sqrt{ \sum_{t=1}^T σ_t^2 } \Big) $ for sufficiently large $T$, where $σ_t^2$ denotes the variance of the rewards at round $t$, $d$ is the dimension of the contexts, and $T$ is the total number of rounds. Furthermore, we introduce a Maximum Likelihood Estimation (MLE)-based algorithm, OFU-MN$^2$L, which achieves an anytime poly(B)-free regret of $O \Big( d \log (BT) \sqrt{ \sum_{t=1}^T σ_t^2 } \Big) $.

强化学习在线学习置信边界贝叶斯优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。