提出最优算法,利用非线性提升多选强化学习的决策效率。
Enjoying Non-linearity in Multinomial Logistic Bandits: A Minimax-Optimal Algorithm
- 扩展非线性常数κ_*至多分类场景,捕捉逻辑回归的非线性特性
- 新算法实现近似最优的$ ilde{O}(Rd oot{KT/κ_*})$累积误差
- 适用于推荐系统等多选项复杂决策任务,理论性能达下界
我们研究多分类逻辑回归老虎机问题,其中学习者通过选择动作最大化期望回报,并基于多个可能结果的概率反馈进行交互。在二分类情形下,近期工作关注逻辑模型非线性的影响力(Faury et al., 2020; Abeille et al., 2021),引入了一个依赖于问题的常数 $κ_* ≥ 1$,其值可能随某些参数指数级增长,由符号函数导数刻画,用于改进 $ ilde{O}(d oot{T})$ 的累计遗憾至 $ ilde{O}(d oot{T/κ_*})$,其中 $d$ 为参数空间维度。本文将该分析拓展至有限动作空间的多分类逻辑回归框架,适用于超过两个选择的复杂应用,如强化学习或推荐系统。为此,我们将 $κ_*$ 的定义推广至多分类设置,并提出一种高效算法,充分利用问题的非线性特征。所提方法获得问题依赖型遗憾界 $ ilde{O}(R d oot{KT/κ_*})$,优于现有最佳 $ ilde{O}(RdK oot{T})$ 的界限。此外,我们给出了匹配的 $ ilde{ ext{Ω}}(dR oot{KT/κ_*})$ 下界,证明该算法最小最大最优,且 $κ_*$ 的定义亦为最优。
原文摘要 · Abstract (English)
We consider the multinomial logistic bandit problem in which a learner interacts with an environment by selecting actions to maximize expected rewards based on probabilistic feedback from multiple possible outcomes. In the binary setting, recent work has focused on understanding the impact of the non-linearity of the logistic model (Faury et al., 2020; Abeille et al., 2021). They introduced a problem-dependent constant $κ_* \geq 1$ that may be exponentially large in some problem parameters and which is captured by the derivative of the sigmoid function. It encapsulates the non-linearity and improves existing regret guarantees over $T$ rounds from $\smash{O(d\sqrt{T})}$ to $\smash{O(d\sqrt{T/κ_*})}$, where $d$ is the dimension of the parameter space. We extend their analysis to the multinomial logistic bandit framework with a finite action space, making it suitable for complex applications with more than two choices, such as reinforcement learning or recommender systems. To achieve this, we extend the definition of $ κ_* $ to the multinomial setting and propose an efficient algorithm that leverages the problem's non-linearity. Our method yields a problem-dependent regret bound of order $ \smash{\widetilde{\mathcal{O}}( R d \sqrt{ {KT}/{κ_*}} ) } $, where $R$ denotes the norm of the vector of rewards and $K$ is the number of outcomes. This improves upon the best existing guarantees of order $ \smash{\widetilde{\mathcal{O}}( RdK \sqrt{T} )}$. Moreover, we provide a matching $\smash{ Ω(dR\sqrt{KT/κ_*})}$ lower-bound, showing that our algorithm is minimax-optimal and that our definition of $κ_*$ is optimal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。