让上千智能体的离线多智能体强化学习变得可行，通过分布空间扩散建模。
Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents
- 将轨迹规划从联合轨迹空间转移到分布空间，利用混沌传播降低维度
- 在上千智能体场景下性能最优，尤其在数据质量差时提升显著
- 理论保证生成策略接近均值场纳什均衡，适合大规模博弈场景
基于扩散的规划在单智能体离线强化学习中表现优异，但因联合轨迹空间的维数灾难，难以扩展到多智能体系统。我们提出MF-Diffuser框架，将轨迹规划提升至轨迹分布的Wasserstein空间，其中混沌传播特性使得少量代表性智能体即可捕捉整体动态。该方法采用价值加权混沌熵目标，平衡生成保真度与回报最大化，并引入分层粗到细策略，在去噪过程中逐步扩大智能体群体。我们建立了端到端次优性边界，包含四个可解释项：均值场近似误差为$O(H^2/\\/sqrt{N})$，而离线分布偏移不会随群体规模$N$增长；并证明生成策略是近似均值场纳什均衡，具有明确收敛保证。在三个均值场强化学习基准测试——包括阶段博弈、序列动态和对抗性团队竞赛——上，MF-Diffuser在多数设置中取得最高回报，尤其在次优离线数据和极端规模（$N \geq 10^3$）下增益最为显著。
原文摘要 · Abstract (English)
Diffusion-based planning has achieved strong results in single-agent offline reinforcement learning, yet scaling to many-agent systems remains intractable due to the curse of dimensionality in the joint trajectory space. We introduce MF-Diffuser, a framework that lifts trajectory planning to the Wasserstein space of trajectory distributions, where the propagation of chaos ensures a small representative subset of agents captures the full population dynamics. Our approach features a value-weighted chaotic entropy objective that reconciles generative fidelity with return maximization, and a hierarchical coarse-to-fine strategy that progressively grows the agent population during denoising. We establish end-to-end suboptimality bounds with four interpretable terms, revealing that mean-field approximation error scales as $O(H^2/\sqrt{N})$ while offline distribution shift provably does not grow with population size $N$, and prove the generated policy is an approximate mean-field Nash equilibrium with explicit convergence guarantees. Experiments on three mean-field RL benchmarks -- spanning stage games, sequential dynamics, and adversarial team competition -- show MF-Diffuser achieves the best return in the majority of settings, with the largest gains on suboptimal offline data and at extreme scales ($N \geq 10^3$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。