arXiv:2502.08259cs.LGcs.AI2025-02被引 7

提出新算法,自动平衡悲观与乐观策略,适应不同学习时长。

Balancing optimism and pessimism in offline-to-online learning

  • 结合悲观(LCB)与乐观(UCB)策略,动态调整探索方式。
  • 在任意时间点表现接近最优的单一策略,性能更稳定。
  • 适合无法预知策略使用时长的现实场景,如推荐系统。

我们研究离线到在线学习设置下的随机有限臂老虎机问题。学习者初始仅能使用非自主收集的离线数据,随后逐步与环境交互以最大化累积收益。该设定面临根本矛盾:短期使用时,基于悲观思想的下置信界(LCB)算法更优,可有效对抗被离线数据覆盖的任意策略;而长期使用时,基于乐观思想的上置信界(UCB)算法更优,其收敛速度几乎达到所有在线算法的理论最优。然而,UCB初期过度探索,导致短期表现劣于LCB。因此,若无法预知策略使用时长,应先用LCB,随时间推移逐步转向类UCB策略。本文提出的新算法实现了这一过渡,并证明其在任意时刻的表现近乎等同于更优的LCB或UCB。核心思想具有普适性,预期可推广至多臂老虎机之外的场景。

原文摘要 · Abstract (English)

We consider what we call the offline-to-online learning setting, focusing on stochastic finite-armed bandit problems. In offline-to-online learning, a learner starts with offline data collected from interactions with an unknown environment in a way that is not under the learner's control. Given this data, the learner begins interacting with the environment, gradually improving its initial strategy as it collects more data to maximize its total reward. The learner in this setting faces a fundamental dilemma: if the policy is deployed for only a short period, a suitable strategy (in a number of senses) is the Lower Confidence Bound (LCB) algorithm, which is based on pessimism. LCB can effectively compete with any policy that is sufficiently "covered" by the offline data. However, for longer time horizons, a preferred strategy is the Upper Confidence Bound (UCB) algorithm, which is based on optimism. Over time, UCB converges to the performance of the optimal policy at a rate that is nearly the best possible among all online algorithms. In offline-to-online learning, however, UCB initially explores excessively, leading to worse short-term performance compared to LCB. This suggests that a learner not in control of how long its policy will be in use should start with LCB for short horizons and gradually transition to a UCB-like strategy as more rounds are played. This article explores how and why this transition should occur. Our main result shows that our new algorithm performs nearly as well as the better of LCB and UCB at any point in time. The core idea behind our algorithm is broadly applicable, and we anticipate that our results will extend beyond the multi-armed bandit setting.

强化学习上下文探索策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。