用跳跃式世界模型实现策略组合规划,显著提升长时任务表现。
Compositional Planning with Jumpy World Models
- 构建多时标跳跃世界模型,离线学习策略序列的动态预测。
- 在复杂操作与导航任务中,零样本性能平均提升200%。
- 适合需要长期规划的强化学习研究者与机器人应用开发者。
能够进行时间抽象的规划是智能决策的核心。我们研究代理通过组合预训练策略作为时序扩展动作来解决问题,这些组合可解决单一策略无法完成的复杂任务。然而,由于长时程预测中的误差累积,难以估计策略序列引发的状态访问分布,使得这种组合规划仍具挑战性。受arXiv:2206.08736提出的几何策略组合框架启发,我们通过学习多步动态的预测模型——即跳跃式世界模型——以非监督方式捕捉不同时间尺度下预训练策略引起的状态占据情况。基于arXiv:2503.09817中的时序差分流方法,我们引入一种新的一致性目标,使跨时间尺度的预测保持一致,从而提升长时程预测精度。进一步地,我们展示了如何结合这些生成式预测,估算任意策略序列在不同时间尺度上的价值。实验表明,使用跳跃式世界模型进行组合规划,在多种基础策略上显著提升零样本性能,在长时任务中相对于原始动作规划平均提升200%。
原文摘要 · Abstract (English)
The ability to plan with temporal abstractions is central to intelligent decision-making. Rather than reasoning over primitive actions, we study agents that compose pre-trained policies as temporally extended actions, enabling solutions to complex tasks that no constituent alone can solve. Such compositional planning remains elusive as compounding errors in long-horizon predictions make it challenging to estimate the visitation distribution induced by sequencing policies. Motivated by the geometric policy composition framework introduced in arXiv:2206.08736, we address these challenges by learning predictive models of multi-step dynamics -- so-called jumpy world models -- that capture state occupancies induced by pre-trained policies across multiple timescales in an off-policy manner. Building on Temporal Difference Flows (arXiv:2503.09817), we enhance these models with a novel consistency objective that aligns predictions across timescales, improving long-horizon predictive accuracy. We further demonstrate how to combine these generative predictions to estimate the value of executing arbitrary sequences of policies over varying timescales. Empirically, we find that compositional planning with jumpy world models significantly improves zero-shot performance across a wide range of base policies on challenging manipulation and navigation tasks, yielding, on average, a 200% relative improvement over planning with primitive actions on long-horizon tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。