arXiv:2606.16656cs.LG2026-06

揭示延迟反馈下线性强化学习的最优性能边界。

Near-Optimal Stochastic Linear Bandits with Delay

  • 区分三类延迟模型,分析其对算法表现的影响。
  • 线性结构使延迟惩罚依赖于维度平方根,比多臂赌博机更难。
  • 适合研究在线学习与延迟反馈的算法设计者阅读。

我们研究了在多种延迟模型下的随机线性老虎机问题,并建立了近似最优的后悔界。结果表明,在损失无关延迟(延迟不依赖于实际损失值,但可能依赖于动作)情况下,延迟仅带来加性后悔增量;随机延迟下该增量与期望延迟成正比,对抗性延迟下则与最大未完成观测数成正比,且两者均为无维度依赖,优于现有最优结果。在损失相关延迟下,线性老虎机比多臂赌博机显著更困难:我们证明了上界与下界匹配(忽略对数因子),其延迟惩罚依赖于维度的平方根。在延迟即回报这一特殊情形中,多臂赌博机可实现的仅依赖最优动作延迟的最优后悔界,在线性场景下也无法达成。这些结果精确刻画了延迟反馈如何与线性泛化相互作用。

原文摘要 · Abstract (English)

We study stochastic linear bandits with delayed feedback under several delay models and establish near-optimal regret guarantees. Our results identify when delayed linear bandits exhibit the same qualitative behavior as multi-armed bandits (MAB), and when the linear structure creates fundamentally new challenges. Specifically, (1) for \emph{loss-independent delays}, where the delay does not depend on the realized loss (but potentially depends on the arm), we show that delays incur only an additive regret penalty. Under stochastic delays, this penalty scales with the expected delay, while under adversarial delays, it scales with the maximum number of outstanding observations. Notably, both delay penalties are dimension-free, improving upon the state-of-the-art results; (2) for \emph{loss-dependent delays}, we show that linear bandits are substantially harder than MAB: unlike in MAB, we prove matching (up to log factors) upper and lower bounds in linear bandits, whose delay penalty depends on the square root of the dimension. (3) for the \emph{delay-as-payoff model}, a special case of loss-dependent delay, we show that the optimal MAB guarantee, which depends only on the delay of the optimal arm, is also unattainable in linear bandits. Together, these results provide a sharp characterization of how delayed feedback interacts with linear generalization.

强化学习线性带宽延迟反馈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。