arXiv:2411.12154stat.MLcs.LG2024-11

提出一种高效线性Bandit算法,实现最优探索与可靠推断的平衡。

Tangential Randomization in Linear Bandits (TRAiL): Guaranteed Inference and Regret Bounds

  • 通过切向随机化扰动最优动作,确保探索充分且计算高效
  • 证明后悔上界为O(√T log T),推断质量最小特征值以Ω(√T)增长
  • 揭示后悔与推断质量的权衡关系,适合关注理论性能的研究者

我们提出并分析了TRAiL(线性带宽中的切向随机化),一种针对强凸函数子水平集型动作集的计算高效、后悔最优的强制探索算法。TRAiL通过标准正则化最小二乘估计模型参数,并将对应点估计的收益最大化动作沿凸紧动作集的切平面扰动后投影回集内。利用矩阵鞅的集中不等式,我们证明了在长度为T的时间段内,推断质量(以设计矩阵最小特征值衡量)以高概率呈现Ω(√T)的增长。基于此结果,我们获得了累积后悔的上界:在至少1−1/T的概率下,为O(√T log T)。同时,我们刻画了任意算法在期望后悔上的Ω(√T)极小极大下界,覆盖广泛的动作/参数集和噪声过程。该分析不仅显著扩展了线性带宽中的下界范围,还揭示了一个后悔与推断质量间的权衡:任何具有O(T^α)期望后悔增长的算法,其期望推断质量必须有Ω(T^{1−α})的渐近增长。实验在L^p单位球动作集上显示,该关系在短期可能被违反,但长期仍趋于满足。因此,最小后悔算法必须具备恰到好处的推断速率——过快或过慢都会导致次优后悔增长。

原文摘要 · Abstract (English)

We propose and analyze TRAiL (Tangential Randomization in Linear Bandits), a computationally efficient regret-optimal forced exploration algorithm for linear bandits on action sets that are sublevel sets of strongly convex functions. TRAiL estimates the governing parameter of the linear bandit problem through a standard regularized least squares and perturbs the reward-maximizing action corresponding to said point estimate along the tangent plane of the convex compact action set before projecting back to it. Exploiting concentration results for matrix martingales, we prove that TRAiL ensures a $Ω(\sqrt{T})$ growth in the inference quality, measured via the minimum eigenvalue of the design (regressor) matrix with high-probability over a $T$-length period. We build on this result to obtain an $\mathcal{O}(\sqrt{T} \log(T))$ upper bound on cumulative regret with probability at least $ 1 - 1/T$ over $T$ periods, and compare TRAiL to other popular algorithms for linear bandits. Then, we characterize an $Ω(\sqrt{T})$ minimax lower bound for any algorithm on the expected regret that covers a wide variety of action/parameter sets and noise processes. Our analysis not only expands the realm of lower-bounds in linear bandits significantly, but as a byproduct, yields a trade-off between regret and inference quality. Specifically, we prove that any algorithm with an $\mathcal{O}(T^α)$ expected regret growth must have an $Ω(T^{1-α})$ asymptotic growth in expected inference quality. Our experiments on the $L^p$ unit ball as action sets reveal how this relation can be violated, but only in the short-run, before returning to respect the bound asymptotically. In effect, regret-minimizing algorithms must have just the right rate of inference -- too fast or too slow inference will incur sub-optimal regret growth.

强化学习在线优化理论分析带宽问题

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。