arXiv:2608.17841stat.MLcs.LG2026-08

提出新算法平衡多臂老虎机的后悔值与策略不稳定性。

Toward the Optimal Regret-Instability Trade-off in Multi-Armed Bandits

  • 设计稳定下界UCB算法,通过动态索引与衰减抽样控制波动。
  • 证明后悔与不稳定性乘积至少为T^{3/2}量级,达到理论极限。
  • 适用于追求稳健决策的强化学习场景,尤其关注算法可靠性。

多臂老虎机算法通常以后悔值衡量性能,但相同后悔值可能对应不同运行间的分配差异。本文研究最坏情况后悔值 $\mathcal{R}_{K,T}$ 与不稳定性 $\mathcal{S}_{K,T}$(即最终抽样次数最大标准差)之间的权衡。在有限时间条件下,无需先前渐近分析中的正则性假设,证明了 $\mathcal{R}_{K,T}\mathcal{S}_{K,T}\ge C T^{3/2}$,其中 $C$ 与 $K$、$T$ 无关。提出新型可调算法 Stabilized Lower-Envelope UCB(SLE-UCB),结合运行下界索引与递减抽样稳定器,实现 $\mathcal{R}_{K,T}\mathcal{S}_{K,T}=O(T^{3/2}\log K)$,在 $T$ 上精确匹配下界,在 $K$ 上仅差对数因子。为证明不稳定性,引入离线前缀表示法,消除在线决策路径依赖,并结合单奖励扰动与Efron–Stein不等式控制抽样方差。结果解决了文献中关于后悔-不稳定性前沿的开放问题。

原文摘要 · Abstract (English)

Multi-armed bandit algorithms are evaluated by regret, yet comparable regret can coexist with different allocations across independent runs. We study the trade-off between worst-case regret $\mathcal{R}_{K,T}$ and instability $\mathcal S_{K,T}$, defined as the largest standard deviation of a terminal pull count, for $K$ arms and $T$ rounds. We prove the finite-time lower bound $\mathcal R_{K,T}\mathcal S_{K,T}\ge C T^{3/2}$, where $C$ is independent of $K$ and $T$, under a finite-time regret condition and without the regularity assumptions imposed in the prior asymptotic analysis. We also introduce Stabilized Lower-Envelope UCB (\textup{\textsc{SLE-UCB}}), a new tunable algorithm combining a running lower-envelope index with a decreasing pull-count stabilizer. \textup{\textsc{SLE-UCB}} satisfies $\mathcal R_{K,T}\mathcal S_{K,T}=O(T^{3/2}\log K)$, with an implicit constant independent of $K$ and $T$, matching the lower bound exactly in $T$ and within a logarithmic factor in $K$. To prove the instability bound, we develop a new offline top-prefix representation that removes path dependence from online decisions. Together with single-reward perturbations and the Efron--Stein inequality, this representation controls pull-count variance. Thus, regret and instability depend reciprocally on $K$, while their product has no polynomial dependence on $K$. These results resolve the open question raised in the literature concerning the sharp arm-dependent regret--instability frontier.

多臂老虎机后悔分析算法稳定性强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。