提出两种少更新的多分类强化学习算法,兼顾效率与性能。
Achieving Limited Adaptivity for Multinomial Logistic Bandits
- 设计批处理与低切换两种策略,仅允许有限次策略更新
- 在随机与对抗性上下文中均实现近似√T的后悔界
- 实验表明少更新算法优于频繁更新的先进方法
多分类逻辑回归强化学习近年受到关注,因其能建模多结果决策问题。本文针对实际应用中需限制策略更新次数的挑战,提出两种新算法:B-MNL-CB(批处理范式)和RS-MNL(低切换范式)。前者在上下文随机生成且更新次数Ω(log log T)时,达到˜O(√T)后悔率;后者适用于对抗性上下文,仅需˜O(log T)次策略更新即可实现˜O(√T)后悔率。实验表明,固定更新次数下,本算法性能远超每轮更新的主流方法,具备强实用性。
原文摘要 · Abstract (English)
Multinomial Logistic Bandits have recently attracted much attention due to their ability to model problems with multiple outcomes. In this setting, each decision is associated with many possible outcomes, modeled using a multinomial logit function. Several recent works on multinomial logistic bandits have simultaneously achieved optimal regret and computational efficiency. However, motivated by real-world challenges and practicality, there is a need to develop algorithms with limited adaptivity, wherein we are allowed only $M$ policy updates. To address these challenges, we present two algorithms, B-MNL-CB and RS-MNL, that operate in the batched and rarely-switching paradigms, respectively. The batched setting involves choosing the $M$ policy update rounds at the start of the algorithm, while the rarely-switching setting can choose these $M$ policy update rounds in an adaptive fashion. Our first algorithm, B-MNL-CB extends the notion of distributional optimal designs to the multinomial setting and achieves $\tilde{O}(\sqrt{T})$ regret assuming the contexts are generated stochastically when presented with $Ω(\log \log T)$ update rounds. Our second algorithm, RS-MNL works with adversarially generated contexts and can achieve $\tilde{O}(\sqrt{T})$ regret with $\tilde{O}(\log T)$ policy updates. Further, we conducted experiments that demonstrate that our algorithms (with a fixed number of policy updates) are extremely competitive (and often better) than several state-of-the-art baselines (which update their policy every round), showcasing the applicability of our algorithms in various practical scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。