arXiv:2603.26741cs.CVcs.AI2026-03被引 12

让智能体根据语言指令在视觉中导航,无需目标图像。

Language-Conditioned World Modeling for Visual Navigation

  • 用扩散模型构建语言引导的世界模型,生成未来状态
  • 两种框架:一种生成连贯轨迹,一种泛化能力更强
  • 适合研究语言理解与环境想象的联合建模

我们研究语言条件下的视觉导航(LCVN),即智能体仅凭初始视角观测和自然语言指令完成导航任务。由于无法获取目标图像,智能体必须依赖语言来塑造感知与连续控制,使得语义定位问题尤为困难。我们将此问题建模为基于语言指令的开环轨迹预测,并提出LCVN数据集,包含39,016条轨迹和117,048条人工验证的指令,支持多种环境与指令风格下的可复现研究。基于该数据集,我们开发了两类LCVN框架:第一类结合基于扩散的世界模型LCVN-WM与在世界模型隐空间训练的演员-评论家代理LCVN-AC;第二类采用自回归多模态架构LCVN-Uni,同时预测动作与未来观测。实验表明,前者生成更时序连贯的轨迹,后者在未见环境中泛化能力更优。这些结果表明,在统一任务设置下联合研究语言定位、想象力生成与策略学习具有重要意义,而LCVN为语言条件世界模型的进一步研究提供了坚实基础。代码已开源。

原文摘要 · Abstract (English)

We study language-conditioned visual navigation (LCVN), in which an embodied agent is asked to follow a natural language instruction based only on an initial egocentric observation. Without access to goal images, the agent must rely on language to shape its perception and continuous control, making the grounding problem particularly challenging. We formulate this problem as open-loop trajectory prediction conditioned on linguistic instructions and introduce the LCVN Dataset, a benchmark of 39,016 trajectories and 117,048 human-verified instructions that supports reproducible research across a range of environments and instruction styles. Using this dataset, we develop LCVN frameworks that link language grounding, future-state prediction, and action generation through two complementary model families. The first family combines LCVN-WM, a diffusion-based world model, with LCVN-AC, an actor-critic agent trained in the latent space of the world model. The second family, LCVN-Uni, adopts an autoregressive multimodal architecture that predicts both actions and future observations. Experiments show that these families offer different advantages: the former provides more temporally coherent rollouts, whereas the latter generalizes better to unseen environments. Taken together, these observations point to the value of jointly studying language grounding, imagination, and policy learning in a unified task setting, and LCVN provides a concrete basis for further investigation of language-conditioned world models. The code is available at https://github.com/F1y1113/LCVN.

视觉导航语言理解世界模型扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。