arXiv:2602.07205cs.LGcs.GT2026-02被引 1

新算法自适应对手策略变化,实现更优的在线学习表现。

Online Learning for Uninformed Markov Games: Empirical Nash-Value Regret and Non-Stationarity Adaptation

  • 引入经验纳什值后悔度,更强且能自动适应对手稳定性
  • 无需参数设置,实现最坏情况下 $O(K^{2/3})$、固定对手时 $O(ar{K})$ 的后悔界
  • 适用于对手策略非平稳的强化学习场景,尤其适合博弈学习研究者

本文研究双人未观测马尔可夫博弈中的在线学习问题,其中对手的动作与策略不可观测。已有工作表明,在不依赖于轮次长度 $H$ 指数级代价的前提下,无法实现无外部后悔。为此,研究者转向较弱的纳什值后悔,并提出基于 V-learning 算法,达到 $O(K^{2/3})$ 后悔。然而该结果对问题难度不自适应:即使对手策略固定(此时可实现 $O(ar{K})$ 外部后悔),仍只能保证 $O(K^{2/3})$ 的较差性能。本文首次同时解决上述局限:提出“经验纳什值后悔”这一更强的指标,当对手策略固定时自然退化为外部后悔;并设计一种无需调参的算法,获得 $O(\min \{\sqrt{K} + (CK)^{1/3}, \sqrt{LK}\})$ 的后悔界,其中 $C$ 衡量对手策略方差,$L$ 为策略切换次数,均不超过 $O(K)$。该结果在固定对手和最坏情形之间平滑插值,并通过重新分析基于周期的 V-learning 算法,结合自适应重启机制实现最终目标。

原文摘要 · Abstract (English)

We study online learning in two-player uninformed Markov games, where the opponent's actions and policies are unobserved. In this setting, Tian et al. (2021) show that achieving no-external-regret is impossible without incurring an exponential dependence on the episode length $H$. They then turn to the weaker notion of Nash-value regret and propose a V-learning algorithm with regret $O(K^{2/3})$ after $K$ episodes. However, their algorithm and guarantee do not adapt to the difficulty of the problem: even in the case where the opponent follows a fixed policy and thus $O(\sqrt{K})$ external regret is well-known to be achievable, their result is still the worse rate $O(K^{2/3})$ on a weaker metric. In this work, we fully address both limitations. First, we introduce empirical Nash-value regret, a new regret notion that is strictly stronger than Nash-value regret and naturally reduces to external regret when the opponent follows a fixed policy. Moreover, under this new metric, we propose a parameter-free algorithm that achieves an $O(\min \{\sqrt{K} + (CK)^{1/3},\sqrt{LK}\})$ regret bound, where $C$ quantifies the variance of the opponent's policies and $L$ denotes the number of policy switches (both at most $O(K)$). Therefore, our results not only recover the two extremes -- $O(\sqrt{K})$ external regret when the opponent is fixed and $O(K^{2/3})$ Nash-value regret in the worst case -- but also smoothly interpolate between these extremes by automatically adapting to the opponent's non-stationarity. We achieve so by first providing a new analysis of the epoch-based V-learning algorithm by Mao et al. (2022), establishing an $O(ηC + \sqrt{K/η})$ regret bound, where $η$ is the epoch incremental factor. Next, we show how to adaptively restart this algorithm with an appropriate $η$ in response to the potential non-stationarity of the opponent, eventually achieving our final results.

在线学习博弈论马尔可夫博弈自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。