提出均值导向算法的理论下界,揭示其学习速度极限。
Mean-based algorithms: A lower bound and regret
- 设计两类新算法,适配未知时长与仅限反馈场景
- 首次建立γ_t序列的下界,证明学习速度有理论极限
- 发现均值导向算法可兼具无悔性,降低被利用风险
均值导向算法是一类在在线学习中对平均回报较低动作赋予低概率的算法。近期研究表明,这类算法能收敛至串行非支配动作,近似经济博弈中的纳什均衡。然而实证显示其在仅带子博弈反馈的场景中收敛较慢。本文研究了在未知时间范围且仅有子博弈反馈的情形下,均值导向算法的性能。首次给出了算法定义序列γ_t的理论下界,明确其学习速度的上限。同时提出了两种新算法:一种推广了ε-贪婪策略,另一种将均值导向Exp3扩展至未知时间场景。实验表明,尽管稍慢,该类算法仍能与现有子博弈反馈算法竞争。进一步分析显示,根据γ_t的选择,此类算法与无悔算法存在非平凡交集,证明存在既为均值导向又具无悔性的算法。这为此前研究提出的该类算法易被利用的结论提供了新视角。
原文摘要 · Abstract (English)
Mean-based algorithms are a class of online learning algorithms that assign low probability to actions with low average rewards. Recent work indicates these algorithms converge favorably to serially undominated actions, which approximate Nash equilibria in economic games. However, empirical studies also show slower convergence compared to established algorithms in bandit-feedback scenarios. We study mean-based algorithms when the time horizon is unknown and only bandit feedback is available. In this setting, we provide the first lower bound on the algorithm-defining sequence $γ_t$ that formally establishes a limit on how fast these algorithms can learn. Additionally, we propose two mean-based algorithms: one generalizes $ε$-greedy, and the other extends the mean-based Exp3 to unknown horizons. Our experiments show that mean-based algorithms, although slightly slower, can perform competitively with other bandit-feedback algorithms. We further analyze the relationship to no-regret algorithms. Depending on the choice of $γ_t$, the intersection with no-regret algorithms is non-trivial, and we show that algorithms exist that are both mean-based and no-regret. This adds context to the "exploitability" of this class of algorithms that previous contributions suggest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。