提出新框架,让追逃游戏策略跨图零样本泛化,无需重训。
Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games
- 在不同图结构上训练时,用对手均衡策略作为指导信号。
- 在未见过的图上测试,追逃策略表现接近微调后的顶尖方法。
- 适合需要快速部署的机器人、安防等实时追逃场景。
对抗性博弈中的均衡学习是博弈论与强化学习的重要课题。追逃游戏(PEG)作为机器人和安防领域的典型实例,其精确求解需指数时间。当底层图结构变化时,现有强化学习方法仍需重新计算或微调,耗时且难以满足实时需求。本文提出均衡策略泛化(EPG)框架,可有效学习具备强跨图零样本泛化能力的通用策略。该框架适用于无出口与多出口场景下的追击方与逃避方,首次实现两类泛化能力。核心思想是在多种图结构上训练策略,对抗各图对应的均衡策略。为构建单图均衡策略,提出一种动态规划算法,可证明生成纯策略纳什均衡,并具近似最优时间复杂度。为提升追击者数量扩展性,进一步设计分组机制与序列模型,分别用于联合策略分解。实验表明,在引入均衡引导与跨图训练距离特征后,EPG框架在多种未见真实图上均实现良好零样本性能;在有出口图上,使用均衡启发式训练的追击策略甚至可媲美顶尖方法的微调结果。
原文摘要 · Abstract (English)
Equilibrium learning in adversarial games is an important topic widely examined in the fields of game theory and reinforcement learning (RL). Pursuit-evasion game (PEG), as an important class of real-world games from the fields of robotics and security, requires exponential time to be accurately solved. When the underlying graph structure varies, even the state-of-the-art RL methods require recomputation or at least fine-tuning, which can be time-consuming and impair real-time applicability. This paper proposes an Equilibrium Policy Generalization (EPG) framework to effectively learn a generalized policy with robust cross-graph zero-shot performance. In the context of PEGs, our framework is generally applicable to both pursuer and evader sides in both no-exit and multi-exit scenarios. These two generalizability properties, to our knowledge, are the first to appear in this domain. The core idea of the EPG framework is to train an RL policy across different graph structures against the equilibrium policy for each single graph. To construct an equilibrium oracle for single-graph policies, we present a dynamic programming (DP) algorithm that provably generates pure-strategy Nash equilibrium with near-optimal time complexity. To guarantee scalability with respect to pursuer number, we further extend DP and RL by designing a grouping mechanism and a sequence model for joint policy decomposition, respectively. Experimental results show that, using equilibrium guidance and a distance feature proposed for cross-graph PEG training, the EPG framework guarantees desirable zero-shot performance in various unseen real-world graphs. Besides, when trained under an equilibrium heuristic proposed for the graphs with exits, our generalized pursuer policy can even match the performance of the fine-tuned policies from the state-of-the-art PEG methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。