提出高效算法解决大规模组合半赌场问题,实现最优后悔界。
Optimal and Efficient Contextual Combinatorial Semi-bandits with General Function Approximation

- 通过凸优化采样组合动作,平衡探索与利用
- 达到最小化最大后悔率 $O(\sqrt{m A T \log |\mathcal{F}|})$
- 适用于任意组合结构,适合推荐系统等场景
我们研究具有通用奖励函数近似的上下文组合半赌场(CCSB)问题。每轮中,学习者观察上下文,选择由基础臂组成的组合动作,并获得每个选中臂的奖励;目标是最大化累积奖励。我们提出 SquareCB.Comb,一种计算高效的算法,每轮通过求解凸优化问题采样组合动作,以平衡探索与利用。该算法可扩展至大规模臂集合,且对动作集仅要求每个组合动作最多包含 $m$ 个臂。我们证明其达到最小化最大后悔率 $O(\sqrt{m A T \log |\mathcal{F}|})$,其中 $A$ 为臂数,$m$ 为组合动作最大基数,$T$ 为时间跨度,$\mathcal{F}$ 为奖励函数类。在可实现设定下,该界与更受限的板式推荐设定中基于策略搜索的先进算法相当,同时推广至任意组合结构和一般奖励函数近似。
原文摘要 · Abstract (English)
We study the contextual combinatorial semi-bandit (CCSB) problem with general reward function approximation. At each round, the learner observes a context, selects a combinatorial action consisting of a subset of basic arms, and receives the reward of each selected arm; the goal is to maximize the cumulative reward over time. We propose SquareCB.Comb, a computationally efficient algorithm that, at each round, solves a convex optimization problem to sample a combinatorial action that balances exploration and exploitation. SquareCB.Comb scales to large arm sets and imposes no structural assumptions on the action set beyond a cardinality bound of $m$ on each combinatorial action. We prove that SquareCB.Comb achieves a minimax optimal regret bound of $O(\sqrt{m A T \log |\mathcal{F}|})$, where $A$ is the number of arms, $m$ is the maximum number of arms in a combinatorial action, $T$ is the time horizon, and $\mathcal{F}$ is the reward function class. In the realizable setting, this bound matches the state-of-the-art regret guarantees achieved by policy search-based algorithms in the more restricted slate recommendation settings, while simultaneously generalizing to arbitrary combinatorial action structures and general reward function approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。