用进化算法实现随机乐观策略,让博弈学习更高效
Randomised Optimism via Competitive Co-Evolution for Matrix Games with Bandit Feedback
- 引入进化算法的变异操作实现随机乐观探索
- 理论证明算法达到次线性遗憾,性能媲美确定性方法
- 适合对博弈学习中探索策略创新感兴趣的读者
学习在博弈中是一个机器学习与人工智能中的基础问题,具有广泛的应用。本文研究未知收益矩阵下的双人零和矩阵博弈与贝叶斯反馈场景,其中每位玩家仅能观测自身动作及对应的噪声收益。已有研究提出相关算法,其中 쬱2046( extit{O et al., 2021})证明确定性乐观策略可实现次线性遗憾。然而,随机乐观在矩阵博弈中的潜力尚未被理论探索。本文提出竞争共进化贝叶斯学习( exttt{CoEvoBL}),将进化算法(EAs)引入贝叶斯框架,通过进化变异算子实现随机乐观。理论上证明 exttt{CoEvoBL} 可实现次线性遗憾,性能与基于确定性乐观的方法相当。据我们所知,这是首个对矩阵博弈中进化贝叶斯学习算法的理论遗憾分析。在多种矩阵博弈基准上的实证评估表明, exttt{CoEvoBL} 不仅达成次线性遗憾,且持续优于经典贝叶斯算法,包括 exttt{Exp-Exp}、 exttt{Exp-Exp-NI} 与 exttt{UCB}。结果凸显了进化贝叶斯学习的潜力,尤其是通过进化算法实现随机乐观在博弈设定中的有效性。
原文摘要 · Abstract (English)
Learning in games is a fundamental problem in machine learning and artificial intelligence, with numerous applications~\citep{silver2016mastering,schrittwieser2020mastering}. This work investigates two-player zero-sum matrix games with an unknown payoff matrix and bandit feedback, where each player observes their actions and the corresponding noisy payoff. Prior studies have proposed algorithms for this setting~\citep{o2021matrix,maiti2023query,cai2024uncoupled}, with \citet{o2021matrix} demonstrating the effectiveness of deterministic optimism (e.g., \ucb) in achieving sublinear regret. However, the potential of randomised optimism in matrix games remains theoretically unexplored. We propose Competitive Co-evolutionary Bandit Learning (\coebl), a novel algorithm that integrates evolutionary algorithms (EAs) into the bandit framework to implement randomised optimism through EA variation operators. We prove that \coebl achieves sublinear regret, matching the performance of deterministic optimism-based methods. To the best of our knowledge, this is the first theoretical regret analysis of an evolutionary bandit learning algorithm in matrix games. Empirical evaluations on diverse matrix game benchmarks demonstrate that \coebl not only achieves sublinear regret but also consistently outperforms classical bandit algorithms, including \exptr~\citep{auer2002nonstochastic}, the variant \exptrni~\citep{cai2024uncoupled}, and \ucb~\citep{o2021matrix}. These results highlight the potential of evolutionary bandit learning, particularly the efficacy of randomised optimism via evolutionary algorithms in game-theoretic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。