在只知自己收益的博弈中,提出新方法实现对最优纯策略的逼近,突破传统后悔率下限。
Adversarial Learning in Games with Bandit Feedback: Logarithmic Pure-Strategy Maximin Regret
- 设计新指标‘纯策略最大最小后悔’,针对仅能观测自身收益的场景优化学习效果
- 在未知对手且仅获自身收益的设定下,实现与游戏特性相关的对数级后悔率
- 适用于对抗性学习、在线决策等需低后悔率的实际场景
在零和博弈中学习是博弈论与机器学习中的基本问题。尽管在自对弈或全信息反馈下外部后悔已取得显著进展,但现实应用常要求学习者面对未知对手且仅能获得所选行动的回报(带通反馈)。在此类挑战性环境中,已知外部后悔至少为 Ω(√T)(T 为轮次数),无法避免。为突破此限制,本文研究在带通反馈下的对抗性学习,目标是最小化相对于最大最小纯策略的亏缺——我们称之为“纯策略最大最小后悔”。分析两种带通反馈模型:非知情(仅揭示实际收益)和知情(同时揭示收益与对手动作)。对于非知情情形,证明 Tsallis-INF 算法在正常形式博弈中实现依赖实例的 O(c log T) 回悔,其中参数 c 取决于游戏;并给出信息论下界,证明对 c 依赖的必要性。为克服此难度,引入知情设置下的 Maximin-UCB,得到另一形式为 O(c' log T) 的后悔界,其中游戏相关参数 c' 可远小于 c。最后将结果推广至任意大动作集上的双线性博弈,分别提出 Tsallis-FTRL-SPM 与 Maximin-LinUCB,建立类似对数级游戏相关后悔界。
原文摘要 · Abstract (English)
Learning to play zero-sum games is a fundamental problem in game theory and machine learning. While significant progress has been made in minimizing external regret in the self-play settings or with full-information feedback, real-world applications often force learners to play against unknown, arbitrary opponents and restrict learners to bandit feedback where only the payoff of the realized action is observable. In such challenging settings, it is well-known that $Ω(\sqrt{T})$ external regret is unavoidable (where T is the number of rounds). To overcome this barrier, we investigate adversarial learning in zero-sum games under bandit feedback, aiming to minimize the deficit against the maximin pure strategy -- a metric we term Pure-Strategy Maximin Regret. We analyze this problem under two bandit feedback models: uninformed (only the realized reward is revealed) and informed (both the reward and the opponent's action are revealed). For uninformed bandit learning of normal-form games, we show that the Tsallis-INF algorithm achieves $O(c \log T)$ instance-dependent regret with a game-dependent parameter $c$. Crucially, we prove an information-theoretic lower bound showing that the dependence on c is necessary. To overcome this hardness, we turn to the informed setting and introduce Maximin-UCB, which obtains another regret bound of the form $O(c' \log T)$ for a different game-dependent parameter $c'$ that could potentially be much smaller than $c$. Finally, we generalize both results to bilinear games over an arbitrary, large action set, proposing Tsallis-FTRL-SPM and Maximin-LinUCB for the uninformed and informed setting respectively and establishing similar game-dependent logarithmic regret bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。