提出首个能处理部分可观测的结构化强化学习方法，收敛快且会利用历史信息。
Recurrent Structural Policy Gradient for Partially Observable Mean Field Games
- 用递归结构结合历史信息和公共噪声，提升策略学习效率。
- 相比无模型强化学习，收敛速度提升一个数量级。
- 适合研究大规模群体交互的学者，尤其关注高效算法设计者。
均值场博弈（MFGs）为大规模群体系统中的交互建模提供了理论框架。然而，由于无模型方法方差高，精确方法计算成本大，算法进展受限。近期混合结构方法（HSMs）通过利用低维个体状态与动作空间及已知转移动态，在蒙特卡洛公共噪声采样基础上精确计算期望回报，从而降低方差并保持可扩展性。但现有HSMs未推广至部分可观测场景。本文提出递归结构策略梯度（RSPG），首个支持公开部分信息的、具备历史感知能力的HSM。RSPG在保证学习历史依赖行为的同时，收敛速度比无模型强化学习方法快一个数量级。为促进MFG研究，我们还推出了基于JAX的MFAX框架，支持解析式与基于样本的均值场更新。代码与示例见 https://clarisse-wibault.github.io/rspg/。
原文摘要 · Abstract (English)
Mean Field Games (MFGs) provide a principled framework for modelling interactions in large population systems. However, algorithmic progress has been limited since model-free methods are high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) reduce variance while maintaining tractability by leveraging low-dimensional individual state and action spaces and known transition dynamics to compute the exact expected return conditioned on Monte Carlo rollouts of common noise. However, HSMs have not been extended to partially observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for MFGs with public partial information. RSPG achieves an order-of-magnitude faster convergence than model-free RL methods while learning history-aware behaviour, unlike current HSMs. To facilitate research into MFGs, we also introduce MFAX, our JAX-based framework for MFGs that supports both analytic and sample-based mean-field updates. MFAX and usage examples can be found at https://clarisse-wibault.github.io/rspg/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。