arXiv:2412.06126math.STcs.IT2024-12被引 21

精确分析UCB算法的后悔值,揭示其实际表现与理论差距。

UCB algorithms for multi-armed bandits: Precise regret and adaptive inference

  • 通过确定性方法精确计算每轮选择次数
  • 发现经典后悔公式仅在间隙较大时成立
  • 证明传统置信区间对序列数据仍有效

上置信界(UCB)算法是解决K臂老虎机问题的一类广泛使用的序列算法。尽管过去几十年的研究已深入探讨其渐近最优性和(近)极小极大最优性,但对其后悔行为的精确理解仍不清晰。这一空白不仅阻碍了对其实际算法效率的评估,也限制了顺序数据收集中的统计推断发展。本文通过确定性刻画一个UCB索引算法的臂选择次数 [Lai87, Agr95, ACBF02],连接了精确后悔分析与自适应统计推断这两个基本方面。所得精确后悔公式不仅能准确捕捉有限时间下个别实例中UCB算法的实际行为,还揭示了现有理论在何种情形下仍有意义。特别地,我们证明经典Lai-Robbins后悔公式仅在次优间隙超过σ√(K log T / T)量级时才精确成立。同时显示,最大后悔值与极小极大后悔值相差一个对数因子,从而否定其严格极小极大最优性。该确定性刻画还对自适应统计推断有重要影响。基于[Lai82]的开创性工作,我们证明了UCB算法具备特定稳定性,从而在两种设置下导出定量中心极限定理,包括老虎机设定中未知奖励的经验均值。这些结果具有重要实际意义:为独立同分布数据设计的传统置信集,在序列数据收集下依然有效。

原文摘要 · Abstract (English)

Upper Confidence Bound (UCB) algorithms are a widely-used class of sequential algorithms for the $K$-armed bandit problem. Despite extensive research over the past decades aimed at understanding their asymptotic and (near) minimax optimality properties, a precise understanding of their regret behavior remains elusive. This gap has not only hindered the evaluation of their actual algorithmic efficiency, but also limited further developments in statistical inference in sequential data collection. This paper bridges these two fundamental aspects--precise regret analysis and adaptive statistical inference--through a deterministic characterization of the number of arm pulls for an UCB index algorithm [Lai87, Agr95, ACBF02]. Our resulting precise regret formula not only accurately captures the actual behavior of the UCB algorithm for finite time horizons and individual problem instances, but also provides significant new insights into the regimes in which the existing theory remains informative. In particular, we show that the classical Lai-Robbins regret formula is exact if and only if the sub-optimality gaps exceed the order $σ\sqrt{K\log T/T}$. We also show that its maximal regret deviates from the minimax regret by a logarithmic factor, and therefore settling its strict minimax optimality in the negative. The deterministic characterization of the number of arm pulls for the UCB algorithm also has major implications in adaptive statistical inference. Building on the seminal work of [Lai82], we show that the UCB algorithm satisfies certain stability properties that lead to quantitative central limit theorems in two settings including the empirical means of unknown rewards in the bandit setting. These results have an important practical implication: conventional confidence sets designed for i.i.d. data remain valid even when data are collected sequentially.

强化学习多臂老虎机统计推断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。