用稀疏视频生成实现秒级远距离导航,让智能体只靠简单指令就能探索未知环境。
Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation
- 用视频生成模型实现长时序规划,突破传统语言模型短视局限
- 生成20秒稀疏未来轨迹,推理速度比原版快27倍
- 零样本实测成功率达顶尖基线2.5倍,首次在夜间场景实现远视距导航
为何视觉语言导航必须依赖详细冗长的语言指令?尽管细节有助于决策,却与真实世界导航目标相悖。理想情况下,智能体应仅凭简单高层意图自主探索未知环境。实现这一目标面临重大挑战:超越视野导航(BVN),即在缺乏密集逐步指引的情况下定位遥远未见目标。现有基于大语言模型(LLM)的方法虽能遵循详尽指令,但因依赖短时监督常表现出短视行为。单纯延长监督范围又会破坏LLM训练稳定性。本文发现,视频生成模型天然具备长时序监督优势,能更好对齐语言指令,因而特别适合BVN任务。我们首次将视频生成模型引入该领域。然而,生成数十秒视频的高延迟使其难以部署。为此,提出SparseVideoNav,仅需生成20秒跨度的稀疏未来轨迹,即可实现亚秒级轨迹推理,相较未优化版本提速27倍。大量真实世界零样本实验表明,SparseVideoNav在BVN任务上成功率是当前最优LLM基线的2.5倍,并首次在复杂夜景中实现此类能力。
原文摘要 · Abstract (English)
Why must vision-language navigation be bound to detailed and verbose language instructions? While such details ease decision-making, they fundamentally contradict the goal for navigation in the real-world. Ideally, agents should possess the autonomy to navigate in unknown environments guided solely by simple and high-level intents. Realizing this ambition introduces a formidable challenge: Beyond-the-View Navigation (BVN), where agents must locate distant, unseen targets without dense and step-by-step guidance. Existing large language model (LLM)-based methods, though adept at following dense instructions, often suffer from short-sighted behaviors due to their reliance on short-horimzon supervision. Simply extending the supervision horizon, however, destabilizes LLM training. In this work, we identify that video generation models inherently benefit from long-horizon supervision to align with language instructions, rendering them uniquely suitable for BVN tasks. Capitalizing on this insight, we propose introducing the video generation model into this field for the first time. Yet, the prohibitive latency for generating videos spanning tens of seconds makes real-world deployment impractical. To bridge this gap, we propose SparseVideoNav, achieving sub-second trajectory inference guided by a generated sparse future spanning a 20-second horizon. This yields a remarkable 27x speed-up compared to the unoptimized counterpart. Extensive real-world zero-shot experiments demonstrate that SparseVideoNav achieves 2.5x the success rate of state-of-the-art LLM baselines on BVN tasks and marks the first realization of such capability in challenging night scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。