用信念惯性揭示算法在非平稳博弈中为何会失效。
Fooling Algorithms in Non-Stationary Bandits using Belief Inertia
- 基于历史奖励均值形成信念惯性,导致算法难以响应变化
- 构造对抗实例使经典算法后悔值线性增长,且常数项大
- 适用于分析重启型算法,对非平稳强化学习有指导意义
我们研究分段平稳多臂老虎机中的最坏情况后悔问题。尽管平稳情形的极小极大理论已成熟,但时变环境下的类似极限仍具挑战性。现有下界依赖于稀疏采样论据:长时间不探索允许对手改变奖励,引发巨大后悔。本文提出一种根本不同的方法——信念惯性论据。分析显示,算法通过历史奖励均值编码的信念会产生惯性,阻碍对新证据的响应。我们证明,这种惯性可被利用,构造出误导经典算法(如探索后承诺、ε-贪婪、UCB)的对抗实例,使其在任意参数设置下,即使仅有一个变化点,后悔值仍随时间线性增长,且常数因子显著。我们还将分析扩展至周期性重启以应对非平稳性的算法,证明其最坏情况后悔依然与T呈线性关系。结果表明,利用信念惯性是推导非平稳老虎机中紧致下界的有效方法。
原文摘要 · Abstract (English)
We study the problem of worst case regret in piecewise stationary multi armed bandits. While the minimax theory for stationary bandits is well established, understanding analogous limits in time-varying settings is challenging. Existing lower bounds rely on what we refer to as infrequent sampling arguments, where long intervals without exploration allow adversarial reward changes that induce large regret. In this paper, we introduce a fundamentally different approach based on a belief inertia argument. Our analysis captures how an algorithm's empirical beliefs, encoded through historical reward averages, create momentum that resists new evidence after a change. We show how this inertia can be exploited to construct adversarial instances that mislead classical algorithms such as Explore Then Commit, epsilon greedy, and UCB, causing them to suffer regret that grows linearly with T and with a substantial constant factor, regardless of how their parameters are tuned, even with a single change point. We extend the analysis to algorithms that periodically restart to handle non stationarity and prove that, even then, the worst case regret remains linear in T. Our results indicate that utilizing belief inertia can be a powerful method for deriving sharp lower bounds in non stationary bandits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。