arXiv:2412.18297cs.GTcs.LG2024-12被引 8

提出应对未知对手的最优学习算法,可高效最大化自身收益。

Learning to Play Against Unknown Opponents

  • 用几何工具构建最优学习策略,适配对手收益分布不确定场景
  • 在期望和最坏情况下均实现渐近最优,时间多项式可解
  • 适用于博弈中对手为理性响应者,适合强化学习与博弈论研究

我们研究一个学习代理在重复对抗一般和博弈时的问题,对手是为最大化自身收益而最优响应学习者算法的策略性参与者。学习者知道自己的收益函数,但对对手收益仅知其来自某分布 $\mathcal{D}$。如何设计学习算法以最大化自身总效用?若算法受限于无遗憾学习,我们证明可在多项式时间内构造出渐近最优算法,适用于期望与最坏情况;若不限制,当博弈规模或 $\mathcal{D}$ 支持集大小为常数时,可在输入大小与 $1/\varepsilon$ 的多项式时间内构造出 $\varepsilon$-最优算法。对于极大极小目标(即最大化最小收益),我们构造了每步多项式时间运行且收敛到最优收益的学习算法。所有结果依赖于将学习算法分析转化为对称为“菜单”的几何对象的研究新工具。

原文摘要 · Abstract (English)

We consider the problem of a learning agent who has to repeatedly play a general sum game against a strategic opponent who acts to maximize their own payoff by optimally responding against the learner's algorithm. The learning agent knows their own payoff function, but is uncertain about the payoff of their opponent (knowing only that it is drawn from some distribution $\mathcal{D}$). What learning algorithm should the agent run in order to maximize their own total utility, either in expectation or in the worst-case over $\mathcal{D}$? When the learning algorithm is constrained to be a no-regret algorithm, we demonstrate how to efficiently construct an optimal learning algorithm (asymptotically achieving the optimal utility) in polynomial time for both the in-expectation and worst-case problems, independent of any other assumptions. When the learning algorithm is not constrained to no-regret, we show how to construct an $\varepsilon$-optimal learning algorithm (obtaining average utility within $\varepsilon$ of the optimal utility) for both the in-expectation and worst-case problems in time polynomial in the size of the input and $1/\varepsilon$, when either the size of the game or the support of $\mathcal{D}$ is constant. Finally, for the special case of the maximin objective, where the learner wishes to maximize their minimum payoff over all possible optimizer types, we construct a learner algorithm that runs in polynomial time in each step and guarantees convergence to the optimal learner payoff. All of these results make use of recently developed machinery that converts the analysis of learning algorithms to the study of the class of corresponding geometric objects known as menus.

博弈学习最优策略无遗憾学习对抗博弈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。