M3PO通过融合模型与非模型探索，高效解决多任务强化学习中的样本效率与泛化难题。
M3PO: Massively Multi-Task Model-Based Policy Optimization
- 用隐式世界模型预测任务结果，不重建观测图像，更聚焦控制目标。
- 在多个基准上达到当前最优性能，超越DreamerV3和PPO等方法。
- 适合需要高样本效率与强泛化的多任务强化学习场景。
我们提出大规模多任务模型基策略优化（M3PO），一种可扩展的模型基强化学习框架，旨在解决单任务设置下的样本效率低下和多任务领域中的泛化能力差问题。现有模型基方法如DreamerV3依赖像素级生成模型，忽略控制导向表示；而模型自由方法如PPO则存在高样本复杂度和探索能力弱的问题。M3PO结合隐式世界模型（不进行观测重构）与混合探索策略，融合模型基规划和模型自由不确定性奖励。通过利用模型基与模型自由价值估计间的差异引导探索，消除了先前方法中的偏差-方差权衡，同时通过信任域优化器保持策略更新稳定。M3PO为现有模型基策略优化提供了一种高效且鲁棒的替代方案，在多个基准测试中达到最先进性能。
原文摘要 · Abstract (English)
We introduce Massively Multi-Task Model-Based Policy Optimization (M3PO), a scalable model-based reinforcement learning (MBRL) framework designed to address sample inefficiency in single-task settings and poor generalization in multi-task domains. Existing model-based approaches like DreamerV3 rely on pixel-level generative models that neglect control-centric representations, while model-free methods such as PPO suffer from high sample complexity and weak exploration. M3PO integrates an implicit world model, trained to predict task outcomes without observation reconstruction, with a hybrid exploration strategy that combines model-based planning and model-free uncertainty-driven bonuses. This eliminates the bias-variance trade-off in prior methods by using discrepancies between model-based and model-free value estimates to guide exploration, while maintaining stable policy updates through a trust-region optimizer. M3PO provides an efficient and robust alternative to existing model-based policy optimization approaches and achieves state-of-the-art performance across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。