提出首个无需参数的最优非平稳无穷臂老虎机算法,可自适应追踪关键变化点。
Tracking Most Significant Shifts in Infinite-Armed Bandits
- 设计黑箱转换框架,将有限臂算法转为无参数的无穷臂非平稳算法
- 引入显著变化检测机制,实现依赖实际变化次数的更优后悔上界
- 首次证明奖励上升不增加难度,适合在线学习与动态环境应用
我们研究一种初始奖励均值从池化分布中采样的无穷臂老虎机问题。以往工作多关注平稳奖励(Berry et al., 1997; Wang et al., 2008; Bonald and Proutiere, 2013; Carpentier and Valko, 2015),而更具挑战性的对抗性/非平稳情形仅近期在衰减奖励(rotting)背景下被研究(Kim et al., 2022; 2024)。此前最优后悔上界需依赖非平稳性参数,且仅对某些池化分布正则性成立。本文首次在所有正则性下获得无需参数的最优后悔上界,并放宽了池化分布假设。首先提出一个黑箱方案,可将针对近平稳环境设计的有限臂MAB算法转化为具有最优后悔保证的无参数无穷臂非平稳算法。接着借鉴近期有限臂老虎机中显著变化的概念(Suk & Kpotufe, 2022),研究该问题下的自然显著变化定义。我们证明,通过在黑箱方案中引入随机淘汰变体,可自适应地实现基于显著变化次数的更紧后悔界。其增强速率仅依赖于衰减型非平稳性,展现出有趣现象:奖励上升不增加非平稳性的困难程度。
原文摘要 · Abstract (English)
We study an infinite-armed bandit problem where actions' mean rewards are initially sampled from a reservoir distribution. Most prior works in this setting focused on stationary rewards (Berry et al., 1997; Wang et al., 2008; Bonald and Proutiere, 2013; Carpentier and Valko, 2015) with the more challenging adversarial/non-stationary variant only recently studied in the context of rotting/decreasing rewards (Kim et al., 2022; 2024). Furthermore, optimal regret upper bounds were only achieved using parameter knowledge of non-stationarity and only known for certain regimes of regularity of the reservoir. This work shows the first parameter-free optimal regret bounds for all regimes while also relaxing distributional assumptions on the reservoir. We first introduce a blackbox scheme to convert a finite-armed MAB algorithm designed for near-stationary environments into a parameter-free algorithm for the infinite-armed non-stationary problem with optimal regret guarantees. We next study a natural notion of significant shift for this problem inspired by recent developments in finite-armed MAB (Suk & Kpotufe, 2022). We show that tighter regret bounds in terms of significant shifts can be adaptively attained by employing a randomized variant of elimination within our blackbox scheme. Our enhanced rates only depend on the rotting non-stationarity and thus exhibit an interesting phenomenon for this problem where rising rewards do not factor into the difficulty of non-stationarity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。