提出新算法,将非单调单指标强化学习的最坏误差降至最优水平。
Optimal Regret for Single Index Bandits

- 分两阶段:先用改进估计器找关键方向,再降维为一维问题求解
- 理论证明误差随时间增长为约T^{2/3},显著优于之前的T^{3/4}
- 首次给出上下界匹配,适合对理论精度要求高的研究者
我们研究单指标强化学习问题,其中奖励依赖于高维上下文在未知一维投影上的表现,通过未知奖励函数建模。该模型将线性与广义线性带宽扩展至非参数情形,尤其适用于奖励函数事先未知的情况。尽管单调奖励函数已有最优后悔界,但一般非单调情形仍不清晰,此前最优边界为$ ilde{/mathcal{O}}(T^{3/4})$(在标准有界性和Lipschitz条件下 [Kang et al., 2025])。本文通过提出简单两阶段算法——带置信上界缩放的单指标带宽(ZoomSIB-UCB),首次实现一般情形下最优后悔界。该算法首先利用归一化Stein估计器估计投影方向,再通过离散化将问题转化为一维带宽,最后使用UCB策略。该方法达到$ ilde{ ext{O}}(T^{2/3})$的后悔上界,且无需额外假设,显著优于以往工作。同时,我们建立了匹配的最小最大下界$ ilde{ ext{Ω}}(T^{2/3})$,证明上界几乎紧致。上下界共同刻画了单指标带宽中的后悔行为本质。实验结果进一步验证了方法的有效性与鲁棒性。
原文摘要 · Abstract (English)
We study the $\textit{single-index bandit}$ problem, where rewards depend on an unknown one-dimensional projection of high-dimensional contexts through an unknown reward function. This model extends linear and generalized linear bandits to a nonparametric setting, and is particularly relevant when the reward function is not known in advance. While optimal regret guarantees are known for monotone reward functions, the general non-monotone case remains poorly understood, with the best known bound being $\tilde{\mathcal{O}}(T^{3/4})$ (under standard boundedness and Lipschitz assumptions on the reward function [Kang et al., 2025]). We close this gap by establishing the optimal regret for general single-index bandits. We propose a simple two-phase algorithm, namely, Zoomed Single Index Bandit with Upper Confidence Bound ($\texttt{ZoomSIB-UCB}$), that first estimates the projection direction via a normalized Stein estimator, and then reduces the problem to a one-dimensional bandit using discretization and finally use UCB. This approach achieves a regret of $\tilde{\mathcal{O}}(T^{2/3})$, and improves significantly upon prior work without any additional assumptions. We also prove a matching minimax lower bound of $\tildeΩ(T^{2/3})$, showing that the upper bound is essentially tight. Our upper and lower bounds together provide a sharp characterization of the regret in single-index bandits. Moreover, the empirical results further demonstrate the effectiveness and robustness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。