arXiv:2409.20440cs.LGstat.ML2024-09被引 4

新算法统一解决随机与对抗性多臂赌博机问题,计算快且性能最优。

Optimism in the Face of Ambiguity Principle for Multi-Armed Bandits

  • 用模糊分布下的乐观原则替代固定扰动,实现统一分析。
  • 在多种场景下达到最优后悔值,计算速度比传统方法快1万倍。
  • 适合需要高效、鲁棒决策的在线学习应用,如推荐系统。

FTRL算法在对抗性和随机多臂赌博机问题中表现优异,但每轮需解优化问题,计算成本高。相比之下,FTPL算法通过扰动收益估计实现高效计算,但分析复杂。本文提出一种新FTPL算法,可在两种环境下生成最优策略。该算法具备类似FTRL的统一后悔分析,又保持类FTPL的低计算开销。不同于依赖已知分布独立加性扰动的现有方法,本文允许扰动服从仅知所属集合的模糊分布,并提出“面对不确定性保持乐观”的原则。该框架推广了现有FTPL方法,同时包含多种最优FTRL方法为特例,这是当前FTPL无法实现的。最后,结合离散选择理论设计了一种高效的二分法算法,用于计算乐观采样概率,其速度比标准FTRL快达10^4倍。结果不仅验证了已有猜想,还揭示了扰动对策略影响的本质,建立了FTRL与FTPL之间的映射关系。

原文摘要 · Abstract (English)

Follow-The-Regularized-Leader (FTRL) algorithms often enjoy optimal regret for adversarial as well as stochastic bandit problems and allow for a streamlined analysis. Nonetheless, FTRL algorithms require the solution of an optimization problem in every iteration and are thus computationally challenging. In contrast, Follow-The-Perturbed-Leader (FTPL) algorithms achieve computational efficiency by perturbing the estimates of the rewards of the arms, but their regret analysis is cumbersome. We propose a new FTPL algorithm that generates optimal policies for both adversarial and stochastic multi-armed bandits. Like FTRL, our algorithm admits a unified regret analysis, and similar to FTPL, it offers low computational costs. Unlike existing FTPL algorithms that rely on independent additive disturbances governed by a \textit{known} distribution, we allow for disturbances governed by an \textit{ambiguous} distribution that is only known to belong to a given set and propose a principle of optimism in the face of ambiguity. Consequently, our framework generalizes existing FTPL algorithms. It also encapsulates a broad range of FTRL methods as special cases, including several optimal ones, which appears to be impossible with current FTPL methods. Finally, we use techniques from discrete choice theory to devise an efficient bisection algorithm for computing the optimistic arm sampling probabilities. This algorithm is up to $10^4$ times faster than standard FTRL algorithms that solve an optimization problem in every iteration. Our results not only settle existing conjectures but also provide new insights into the impact of perturbations by mapping FTRL to FTPL.

多臂赌博机在线学习优化算法后悔最小化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。