arXiv:2608.07267cs.AIcs.CV2026-08

让智能体在3D场景中更准导航,靠的是预存几何信息。

WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN

论文配图:WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN
图 1 · 摘自论文原文
  • 用3D几何特征做视觉-动作联合预测,让每步决策都有环境依据
  • 在GN-Bench上闭环导航成功率超基线模型12.7%,路径更优
  • 适合做连续视觉导航的系统研发,尤其关注精准空间理解的场景

近期视觉语言导航(VLN)系统将预训练视觉语言模型(VLM)转化为视觉语言动作(VLA)策略,直接将视角观测与语言指令映射为导航动作。尽管语义能力较强,但此类以动作为中心的训练未显式建模代理预测运动下视觉观测应如何演化。生成式世界动作模型(WAMs)联合预测未来观测与动作,但现有连续VLN的WAMs未将未来视图和动作生成条件化于从观测历史推断出的几何感知表征。我们提出WNM-3D:一种带3D场景条件的生成式世界导航模型,用于连续视觉语言导航。为整合历史观测形成持久场景上下文,采用冻结的前馈几何编码器从单目自指RGB历史中提取几何感知表征,并通过可训练的3D场景到令牌适配器将其转换为世界动作扩散变换器令牌空间中的固定长度前缀。通过块因果注意力,该前缀对每个未来视频-动作块进行条件化,为未来视图和动作生成提供共享几何上下文。我们通过在A*生成示范上监督微调、在策略访问状态上采用DAgger式适应,以及在闭环执行中使用反事实舞蹈GRPO优化来训练WNM-3D。在GN-Bench上的实验表明,WNM-3D在闭环导航中优于强基线的VLM导航策略及其2D条件化对应模型。阶段消融实验进一步显示,DAgger-SFT带来更大的成功率达提升,而反事实舞蹈GRPO则随后同时改善导航成功率与路径效率。

原文摘要 · Abstract (English)

Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions. Although semantically capable, such action-centric training does not explicitly model how the agent's visual observations should evolve under its predicted motion. Generative world-action models (WAMs) jointly predict future observations and actions, yet existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observed history. We present WNM-3D, a generative World Navigation Model with 3D scene conditioning for continuous VLN. To consolidate past observations into persistent scene context, a frozen feed-forward geometry encoder extracts geometry-aware representations from the monocular egocentric RGB history, and a trainable 3D Scene-to-Token Adapter converts them into a fixed-length prefix in the token space of the world-action Diffusion Transformer. Through block-causal attention, this prefix conditions every future video-action block, providing a shared geometric context for both future-view and action generation. We train WNM-3D through supervised world-action fine-tuning on A*-generated demonstrations, DAgger-style adaptation on policy-visited states, and Counterfactual DanceGRPO refinement for closed-loop execution. Experiments on GN-Bench show that WNM-3D outperforms strong VLM-based navigation policies and its 2D-conditioned counterpart in closed-loop navigation. Stage-wise ablations further show that DAgger-SFT provides the larger success-rate gain, while Counterfactual DanceGRPO subsequently improves both navigation success and path efficiency.

视觉导航3D场景扩散模型闭环控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。