为FTPL设计自适应学习率,实现更好在线学习性能。
Adaptive Learning Rates with Surrogate Probability for Follow-the-Perturbed-Leader
- 用代理概率函数替代真实概率,实现无需优化的自适应学习率
- 在任意α>1时,对帕累托扰动的FTPL给出最优双世界保证
- 保持计算简单,适用于带专家建议的多臂赌博机问题
Follow-the-regularized-leader框架在在线学习中表现出色,学习率选择至关重要。近期基于动作选择概率设计的自适应学习率(通过凸优化求解)已在多种赌博机问题中取得改进的最优双世界(BOBW)保证。相比之下,其计算高效的替代方法——跟随扰动领袖(FTPL)由于无优化特性,难以设计依赖概率的自适应学习率,导致其BOBW保证相对有限。为此,本文提出一种基于代理概率函数的自适应学习率,仅需可用数据即可计算,无需真实概率。基于此学习率,我们为带有帕累托扰动的FTPL在任意形状参数α>1下建立了BOBW保证,推广了以往仅限于α=2的结论。进一步证明了在带专家建议的赌博机问题中,使用自适应学习率的FTPL仍具BOBW性质。该方法在保持FTPL计算简洁性的同时实现概率依赖的自适应性,其代理函数策略对其他算法框架亦具独立价值。
原文摘要 · Abstract (English)
Follow-the-regularized-leader framework has shown effectiveness and flexibility in online learning problems, where the choice of learning rates are known to be crucial. Recently, adaptive learning rates defined in terms of the arm-selection probabilities, obtained by solving convex optimization, have achieved improved best-of-both-worlds (BOBW) guarantees in various bandit problems. In contrast, BOBW guarantees for its computationally efficient alternative, follow-the-perturbed-leader (FTPL), remain relatively limited since its optimization-free nature ironically makes the design of adaptive, probability-dependent learning rates non-trivial. To address this challenge, we propose an adaptive learning rate for FTPL by introducing surrogate probability functions that can be computed only from the available quantities, without requiring the exact probabilities. Based on these learning rates with surrogate functions, we provide the BOBW guarantee for FTPL with Pareto perturbations for any shape parameter $α>1$, generalizing prior results restricted to specific choices of $α=2$. We further show the BOBW guarantees for FTPL with adaptive learning rates in the bandit problem with expert advices. Our approach preserves the computational simplicity of FTPL while enabling probability-dependent adaptivity, and the surrogate-based methodology may be of independent interest in other algorithmic frameworks beyond FTPL and learning rate designs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。