提出可同时优化物理合理性、动作一致性和视觉保真度的强化学习框架。
PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration
- 基于预训练世界模型，通过奖励驱动优化PAV三大目标。
- 在多个基准上平均提升5.6%，显著改善动作泛化能力。
- 适合需要可靠策略评估的机器人仿真与具身智能研究者。
动作条件世界模型是具身人工智能的关键组件，作为可扩展的策略评估器，能减少对昂贵真实世界轨迹的依赖。为准确捕捉多样动作引发的动力学变化，此类模型需同时满足物理合理性（P）、动作一致性（A）和视觉保真度（V），统称为PAV，并在分布内（ID）专家示范和分布外（OOD）动作下保持鲁棒性。然而，现有方法主要依赖于分布内动作-视频对与像素级重建损失，未显式优化PAV目标，且在专家数据之外泛化能力差。为此，我们提出PAVXploreRL，一个基于预训练潜在世界模型的强化学习框架，通过奖励驱动训练显式优化PAV目标。为提升动作泛化能力，该方法联合使用分布内轨迹与噪声驱动的分布外动作探索，无需配对视频监督。实验表明，PAVXploreRL持续优于预训练基线，在多个基准上实现平均5.6%的提升，并生成更高质量的PAV特性。作为策略评估器，其还能提供更可靠的性能估计，降低以往仅依赖专家数据的世界模型产生的过估计偏差，如Ctrl-World。代码已开源：https://github.com/Social-AI-Studio/PAVXploreRL。
原文摘要 · Abstract (English)
Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV-while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World. Code: https://github.com/Social-AI-Studio/PAVXploreRL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。