arXiv:2603.25981cs.ROcs.AI2026-03被引 3

用策略引导世界模型规划,让机器人更准地听懂指令找路。

Policy-Guided World Model Planning for Language-Conditioned Visual Navigation

  • 先用策略生成动作分布,再以此初始化世界模型规划
  • 在真实任务中达到92.3%成功率,优于单一策略或盲目规划
  • 适合需要长距离、高精度导航的机器人应用

给定自然语言指令的视觉目标导航仍是具身智能的核心挑战。现有方法要么依赖反应式策略难以处理长程规划,要么使用世界模型在高维空间中动作初始化不佳。我们提出PiJEPA,一种两阶段框架,结合学习型导航策略与潜在世界模型规划,实现语言条件下的视觉导航。第一阶段,在CAST数据集上微调基于Octo的通用策略,并融合冻结的预训练视觉编码器(DINOv2或V-JEPA-2),生成基于当前观测和语言指令的动作分布。第二阶段,利用该策略分布作为初始条件,对独立训练的JEPA世界模型进行模型预测路径积分(MPPI)规划,该模型在相同编码器的嵌入空间中预测未来潜在状态。通过从策略先验而非无信息高斯分布初始化采样,规划器收敛更快,生成更优动作序列以达成目标。系统性评估了视觉编码器骨干(DINOv2 vs V-JEPA-2)在策略与世界模型中的影响。真实世界导航实验表明,PiJEPA显著优于纯策略执行与无引导世界模型规划,提升目标达成率与指令遵循准确率。

原文摘要 · Abstract (English)

Navigating to a visually specified goal given natural language instructions remains a fundamental challenge in embodied AI. Existing approaches either rely on reactive policies that struggle with long-horizon planning, or employ world models that suffer from poor action initialization in high-dimensional spaces. We present PiJEPA, a two-stage framework that combines the strengths of learned navigation policies with latent world model planning for instruction-conditioned visual navigation. In the first stage, we finetune an Octo-based generalist policy, augmented with a frozen pretrained vision encoder (DINOv2 or V-JEPA-2), on the CAST navigation dataset to produce an informed action distribution conditioned on the current observation and language instruction. In the second stage, we use this policy-derived distribution to warm-start Model Predictive Path Integral (MPPI) planning over a separately trained JEPA world model, which predicts future latent states in the embedding space of the same frozen encoder. By initializing the MPPI sampling distribution from the policy prior rather than from an uninformed Gaussian, our planner converges faster to high-quality action sequences that reach the goal. We systematically study the effect of the vision encoder backbone, comparing DINOv2 and V-JEPA-2, across both the policy and world model components. Experiments on real-world navigation tasks demonstrate that PiJEPA significantly outperforms both standalone policy execution and uninformed world model planning, achieving improved goal-reaching accuracy and instruction-following fidelity.

视觉导航世界模型策略规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。