arXiv:2605.25789cs.LGcs.AI2026-05

免费探索阶段可显著降低多臂赌博机的累计后悔,算法更智能地利用初始探索机会。

On the Benefits of Free Exploration for Regret Minimization in Multi-Armed Bandits

论文配图:On the Benefits of Free Exploration for Regret Minimization in Multi-Armed Bandits
图 1 · 摘自论文原文
  • 设计两阶段算法:先自由探索,再自适应减少后悔
  • 在对数级探索预算下,后悔量明显低于无免费探索的算法
  • 适用于需要高效探索策略的强化学习场景

我们研究了一种随机多臂赌博机问题,其中代理在后悔开始累积前可获得一次免费探索预算,这一设置无法被经典后悔最小化或纯探索范式涵盖。目标是设计一种自适应策略,在初始免费探索阶段战略性地探索环境,并最小化后续阶段的累计后悔。我们形式化了带免费探索的后悔最小化问题,并识别出一个关键区间:免费探索预算随时间范围对数增长。为量化高概率下因免费探索带来的后悔节省,我们引入了一类新型策略,称为(α,β)-可能节省策略。提出一种两阶段、可能节省算法UFE-KLUCB-H,包含一个有原则的免费探索策略UFE和一个历史感知的后悔最小化策略KLUCB-H。推导出实例依赖的上界,表明UFE-KLUCB-H的后悔严格低于无免费探索访问权限的策略。同时,基于针对免费探索设置的新型多实例扰动论证,建立实例依赖的下界,证明对于双值赌博机,UFE-KLUCB-H近乎最优。上下界揭示了积累后悔量随可用免费探索量变化的显著相变。模拟结果表明,强制探索与算法自适应性共同带来更大后悔节省。

原文摘要 · Abstract (English)

We study a stochastic multi-armed bandit problem where an agent is granted a free exploration budget before regret accumulates, a setting not captured by the classic regret minimization or pure exploration paradigms. The goal is to design an adaptive policy that strategically explores the bandit instance in the initial free exploration phase and minimizes the cumulative regret in the subsequent phase. We formalize this regret minimization with free exploration problem and identify an interesting regime where the free exploration budget scales logarithmically with the time horizon. To quantify the amount of regret saved with high probability as a result of the availability of the free exploration phase, we introduce a novel set of policies known as $(α,β)$-probably saving policies. We propose a two-phase, probably saving algorithm, UFE-KLUCB-H, which consists of a principled free exploration policy, UFE, and a history-aware regret minimization policy KLUCB-H. Instance-dependent upper bounds on UFE-KLUCB-H are derived, showing that UFE-KLUCB-H accumulates strictly less regret than policies that do not have access to a free exploration phase. Complementarily, we derive instance-dependent lower bounds based on novel multi-instance perturbation arguments tailored to the free-exploration setting, demonstrating the near-optimality of UFE-KLUCB-H for two-valued bandits. Our upper and lower bounds reveal sharp phase transitions in the accumulated regret depending on the amount of available free exploration. Simulations are conducted to demonstrate that forced exploration and adaptivity in the algorithm lead to greater regret savings.

强化学习多臂赌博机后悔最小化探索策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。