arXiv:2508.06269cs.LGcs.AI2025-08被引 4

提出OM2P,让离线多智能体生成策略一步采样,更快更省显存。

OM2P: Offline Multi-Agent Mean-Flow Policy

  • 用均流匹配+Q值监督,解决生成目标与奖励最大化的不一致。
  • 训练速度提升10.8倍,显存占用减少3.8倍,适合资源受限场景。
  • 首个成功将均流模型用于离线多智能体学习,适合协作任务研究者。

生成模型(尤其是扩散和流模型)在离线多智能体强化学习中表现潜力巨大,但其迭代生成过程导致采样效率低,在时间敏感或资源受限场景下不实用。为此,我们提出OM2P(Offline Multi-Agent Mean-Flow Policy),一种新型离线MARL算法,实现高效的一步动作采样。为解决生成目标与奖励最大化之间的偏差,引入奖励感知优化方案,结合精心设计的均流匹配损失与Q函数监督。此外,设计广义时间步分布和无导数估计策略,降低内存开销并提升训练稳定性。在Multi-Agent Particle和MuJoCo基准上的实证评估表明,OM2P性能优越,显存使用最多减少3.8倍,训练时间最快提升10.8倍。本方法首次成功将均流模型集成到离线MARL中,为协作多智能体场景下的实用、可扩展生成策略铺平道路。

原文摘要 · Abstract (English)

Generative models, especially diffusion and flow-based models, have been promising in offline multi-agent reinforcement learning. However, integrating powerful generative models into this framework poses unique challenges. In particular, diffusion and flow-based policies suffer from low sampling efficiency due to their iterative generation processes, making them impractical in time-sensitive or resource-constrained settings. To tackle these difficulties, we propose OM2P (Offline Multi-Agent Mean-Flow Policy), a novel offline MARL algorithm to achieve efficient one-step action sampling. To address the misalignment between generative objectives and reward maximization, we introduce a reward-aware optimization scheme that integrates a carefully-designed mean-flow matching loss with Q-function supervision. Additionally, we design a generalized timestep distribution and a derivative-free estimation strategy to reduce memory overhead and improve training stability. Empirical evaluations on Multi-Agent Particle and MuJoCo benchmarks demonstrate that OM2P achieves superior performance, with up to a 3.8x reduction in GPU memory usage and up to a 10.8x speed-up in training time. Our approach represents the first to successfully integrate mean-flow model into offline MARL, paving the way for practical and scalable generative policies in cooperative multi-agent settings.

多智能体生成模型离线RL均流

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。