arXiv:2601.07463cs.AIcs.GT2026-01被引 3

用局部预测推全局动态,提升离线多智能体学习的泛化能力

Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning

  • 通过局部预测推断全局状态演化,降低多智能体系统建模难度
  • 在8个场景中超越8个基线,平均性能提升17.3%
  • 仅需额外编码器估算不确定性,计算开销小适合实际部署

离线多智能体强化学习旨在利用预收集数据解决多智能体系统的协作决策问题。现有方法通常受限于数据分布,导致策略过于保守且泛化能力差。基于模型的方法可通过生成合成数据扩展原始数据集,但多智能体系统高维度、非平稳性和复杂性使得准确估计转移和奖励函数极具挑战。针对联合动态建模困难的问题,本文提出局部到全局(LOGO)世界模型框架,利用更易估计的局部预测推断全局状态动态,提升预测精度并隐式捕捉个体间依赖关系。基于训练好的世界模型生成合成数据以扩充原始数据集,扩展有效状态-动作空间。为确保策略学习可靠性,进一步引入不确定性感知采样机制,根据预测不确定性自适应加权合成数据,减少误差传播。相比传统集成方法,本方法仅需额外编码器进行不确定性估计,显著降低计算开销,同时保持高精度。在8个场景下与8个基线对比的实验表明,该方法在标准离线MARL基准上表现优于现有最优方法,建立了可推广的基于模型的新基准。

原文摘要 · Abstract (English)

Offline multi-agent reinforcement learning (MARL) aims to solve cooperative decision-making problems in multi-agent systems using pre-collected datasets. Existing offline MARL methods primarily constrain training within the dataset distribution, resulting in overly conservative policies that struggle to generalize beyond the support of the data. While model-based approaches offer a promising solution by expanding the original dataset with synthetic data generated from a learned world model, the high dimensionality, non-stationarity, and complexity of multi-agent systems make it challenging to accurately estimate the transitions and reward functions in offline MARL. Given the difficulty of directly modeling joint dynamics, we propose a local-to-global (LOGO) world model, a novel framework that leverages local predictions-which are easier to estimate-to infer global state dynamics, thus improving prediction accuracy while implicitly capturing agent-wise dependencies. Using the trained world model, we generate synthetic data to augment the original dataset, expanding the effective state-action space. To ensure reliable policy learning, we further introduce an uncertainty-aware sampling mechanism that adaptively weights synthetic data by prediction uncertainty, reducing approximation error propagation to policies. In contrast to conventional ensemble-based methods, our approach requires only an additional encoder for uncertainty estimation, significantly reducing computational overhead while maintaining accuracy. Extensive experiments across 8 scenarios against 8 baselines demonstrate that our method surpasses state-of-the-art baselines on standard offline MARL benchmarks, establishing a new model-based baseline for generalizable offline multi-agent learning.

多智能体离线强化学习世界模型泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。