用自回归模型预测空中导航的环境变化并生成动作,提升导航成功率。
WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation

- 基于自回归框架预测短时环境状态变化,直接输出可执行路径点。
- 在公开数据集上比现有方法成功率提升12%以上,复杂场景优势更明显。
- 首次为自回归世界动作模型设计强化学习优化方法,支持零样本部署到真实无人机。
空中视觉语言导航(Aerial VLN)要求智能体在三维环境中通过闭环感知与动作,遵循自然语言指令前进。本文提出世界动作建模新范式:将空中VLP视为预测驱动的世界-动作问题,即智能体应预判潜在环境演化,并依据预测结果行动。为此,我们提出首个用于空中VLN的自回归世界动作模型(WorldVLN)。不同于生成完整视频序列的世界模型,WorldVLN采用潜在自回归视频骨干网络,预测短时程世界状态转移,并直接解码为可执行的路径点动作。每次动作执行后,新观测被编码回自回归上下文,实现闭环预测。我们还设计了两阶段训练框架:首先将视频先验锚定于指令条件下的导航动态;再引入面向动作的GRPO(Action-aware GRPO),这是首个专为自回归世界动作模型设计的强化学习方法,通过下游轨迹后果优化路径决策。在公开的室内外基准上,WorldVLN持续优于现有视觉-语言-动作基线,成功率提升12%以上,复杂任务中优势更显著。该模型还可零样本迁移到真实无人机部署,表明其为空间动作任务提供了有前景的新路径。演示与代码已开源。
原文摘要 · Abstract (English)
Aerial vision-language navigation (VLN) requires agents to follow natural-language instructions through closed-loop perception and action in 3D environments. We argue that aerial VLN can be formulated as a prediction-driven world-action problem: the agent should anticipate latent world evolution and act according to the predicted consequences. To this end, we propose WorldVLN, the first autoregressive world action model for aerial VLN. Unlike full-sequence video-generation world models that generate an entire visual clip, WorldVLN adapts a latent autoregressive video backbone to predict short-horizon world-state transitions and directly decodes them into executable waypoint actions. After each action segment is executed, newly received observations are encoded back into the autoregressive context, enabling closed-loop world-action prediction. We further introduce a two-stage training framework that first grounds the video prior in instruction-conditioned navigation dynamics and then develops Action-aware GRPO, the first reinforcement learning method tailored to autoregressive WAMs, to optimize waypoint decisions through their downstream rollout consequences. On public outdoor and indoor benchmarks, WorldVLN consistently outperforms existing Vision-Language-Action baselines with 12\%+ success-rate gains and larger advantages on challenging cases. It further transfers zero-shot to real drone deployment, suggesting that the proposed WorldVLN offers a promising route for spatial action tasks. Demos and code are available at https://embodiedcity.github.io/WorldVLN/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。