arXiv:2605.09454stat.MLcs.LG2026-05

提出新算法,将非单调单指标强化学习的最坏误差降至最优水平。

Optimal Regret for Single Index Bandits

论文配图:Optimal Regret for Single Index Bandits
图 1 · 摘自论文原文
  • 分两阶段:先用改进估计器找关键方向,再降维为一维问题求解
  • 理论证明误差随时间增长为约T^{2/3},显著优于之前的T^{3/4}
  • 首次给出上下界匹配,适合对理论精度要求高的研究者

我们研究单指标强化学习问题,其中奖励依赖于高维上下文在未知一维投影上的表现,通过未知奖励函数建模。该模型将线性与广义线性带宽扩展至非参数情形,尤其适用于奖励函数事先未知的情况。尽管单调奖励函数已有最优后悔界,但一般非单调情形仍不清晰,此前最优边界为$ ilde{/mathcal{O}}(T^{3/4})$(在标准有界性和Lipschitz条件下 [Kang et al., 2025])。本文通过提出简单两阶段算法——带置信上界缩放的单指标带宽(ZoomSIB-UCB),首次实现一般情形下最优后悔界。该算法首先利用归一化Stein估计器估计投影方向,再通过离散化将问题转化为一维带宽,最后使用UCB策略。该方法达到$ ilde{ ext{O}}(T^{2/3})$的后悔上界,且无需额外假设,显著优于以往工作。同时,我们建立了匹配的最小最大下界$ ilde{ ext{Ω}}(T^{2/3})$,证明上界几乎紧致。上下界共同刻画了单指标带宽中的后悔行为本质。实验结果进一步验证了方法的有效性与鲁棒性。

原文摘要 · Abstract (English)

We study the $\textit{single-index bandit}$ problem, where rewards depend on an unknown one-dimensional projection of high-dimensional contexts through an unknown reward function. This model extends linear and generalized linear bandits to a nonparametric setting, and is particularly relevant when the reward function is not known in advance. While optimal regret guarantees are known for monotone reward functions, the general non-monotone case remains poorly understood, with the best known bound being $\tilde{\mathcal{O}}(T^{3/4})$ (under standard boundedness and Lipschitz assumptions on the reward function [Kang et al., 2025]). We close this gap by establishing the optimal regret for general single-index bandits. We propose a simple two-phase algorithm, namely, Zoomed Single Index Bandit with Upper Confidence Bound ($\texttt{ZoomSIB-UCB}$), that first estimates the projection direction via a normalized Stein estimator, and then reduces the problem to a one-dimensional bandit using discretization and finally use UCB. This approach achieves a regret of $\tilde{\mathcal{O}}(T^{2/3})$, and improves significantly upon prior work without any additional assumptions. We also prove a matching minimax lower bound of $\tildeΩ(T^{2/3})$, showing that the upper bound is essentially tight. Our upper and lower bounds together provide a sharp characterization of the regret in single-index bandits. Moreover, the empirical results further demonstrate the effectiveness and robustness of our approach.

强化学习带宽理论最优后悔

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。