通过慢速语言模型与快速图对齐协同,实现高效零样本视觉语言导航。
SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment
- 慢速规划生成子目标链,快速反应执行实时导航。
- 相比之前方法,轨迹总令牌消耗减少50%以上,速度提升3.5倍。
- 支持按内部置信度异步触发大模型,适合真实机器人部署。
近期大型视觉语言模型(VLM)和大型语言模型(LLM)的发展推动了零样本视觉语言导航(VLN)方法的进展,使智能体仅通过自身感知与推理即可遵循自然语言指令。然而,现有零样本方法通常构建简单观测图,并在每一步执行VLM-LLM推理,导致延迟高、计算成本大,限制了实时应用。为此,我们提出SFCo-Nav,一种受慢-快认知协作启发的高效零样本VLN框架。该框架包含三个模块:1)基于慢速LLM的规划器,生成一系列与想象物体图关联的子目标;2)快速反应导航器,实时构建物体图并执行子目标;3)轻量级异步慢-快桥,对齐结构化、带属性的想象图与感知图,估计导航置信度,仅在必要时触发慢速LLM规划器。据我们所知,SFCo-Nav是首个支持根据内部置信度异步触发的慢-快协同零样本VLN系统。在公开的R2R和REVERIE基准上评估,SFCo-Nav达到或超过现有最优零样本方法的成功率,同时将每条轨迹的总令牌消耗降低超50%,运行速度提升3.5倍以上。最后,我们在酒店套房中的四足机器人上展示了其效率与实用性。
原文摘要 · Abstract (English)
Recent advances in large vision-language models (VLMs) and large language models (LLMs) have enabled zero-shot approaches to visual language navigation (VLN), where an agent follows natural language instructions using only ego perception and reasoning. However, existing zero-shot methods typically construct a naive observation graph and perform per-step VLM-LLM inference on it, resulting in high latency and computation costs that limit real-time deployment. To address this, we present SFCo-Nav, an efficient zero-shot VLN framework inspired by the principle of slow-fast cognitive collaboration. SFCo-Nav integrates three key modules: 1) a slow LLM-based planner that produces a strategic chain of subgoals, each linked to an imagined object graph; 2) a fast reactive navigator for real-time object graph construction and subgoal execution; and 3) a lightweight asynchronous slow-fast bridge aligns advanced structured, attributed imagined and perceived graphs to estimate navigation confidence, triggering the slow LLM planner only when necessary. To the best of our knowledge, SFCo-Nav is the first slow-fast collaboration zero-shot VLN system supporting asynchronous LLM triggering according to the internal confidence. Evaluated on the public R2R and REVERIE benchmarks, SFCo-Nav matches or exceeds prior state-of-the-art zero-shot VLN success rates while cutting total token consumption per trajectory by over 50% and running more than 3.5 times faster. Finally, we demonstrate SFCo-Nav on a legged robot in a hotel suite, showcasing its efficiency and practicality in indoor environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。