arXiv:2412.00798cs.LGstat.ML2024-12

解决组合奖励随时间累积的强化学习问题,提升推荐与机器人系统决策效率。

Combinatorial Rising Bandits

  • 提出新型组合上升贝叶斯算法,捕捉基臂对后续奖励的累积影响。
  • 在真实与合成环境中验证算法性能,理论证明其后悔上界紧致。
  • 适用于需长期反馈积累的场景,如社交推荐与机器人学习。

组合在线学习是通过序列交互从基础臂中选择最优组合动作(超臂)以获取随机奖励的核心任务,广泛应用于机器人、社交广告、网络路由和推荐系统。现实中常出现奖励上升现象:执行一个基础臂不仅带来即时收益,还提升未来奖励,例如机器人通过练习进步,或成功推荐历史增强社交影响力。关键在于,这种提升可能传播至共享相同基础臂的多个超臂,形成现有贝叶斯模型无法涵盖的依赖关系。为此,我们提出组合上升贝叶斯(CRB)框架,并设计一种可证明高效且实证有效的算法——组合上升置信上界(CRUCB)。我们在真实深度强化学习环境与合成场景中实证验证了CRUCB的有效性,理论分析建立了紧致的后悔上界。二者共同体现了该方法的实践价值与理论严谨性。代码已公开于 https://github.com/ml-postech/Combinatorial-Rising-Bandits。

原文摘要 · Abstract (English)

Combinatorial online learning is a fundamental task for selecting the optimal action (or super arm) as a combination of base arms in sequential interactions with systems providing stochastic rewards. It is applicable to diverse domains such as robotics, social advertising, network routing, and recommendation systems. In many real-world scenarios, we often encounter rising rewards, where playing a base arm not only provides an instantaneous reward but also contributes to the enhancement of future rewards, e.g., robots improving through practice and social influence strengthening in the history of successful recommendations. Crucially, these enhancements may propagate to multiple super arms that share the same base arms, introducing dependencies beyond the scope of existing bandit models. To address this gap, we introduce the Combinatorial Rising Bandit (CRB) framework and propose a provably efficient and empirically effective algorithm, Combinatorial Rising Upper Confidence Bound (CRUCB). We empirically demonstrate the effectiveness of CRUCB in realistic deep reinforcement learning environments and synthetic settings, while our theoretical analysis establishes tight regret bounds. Together, they underscore the practical impact and theoretical rigor of our approach. Our code is available at https://github.com/ml-postech/Combinatorial-Rising-Bandits.

强化学习组合优化贝叶斯学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。