用视觉想象提升大模型导航能力,无需地图也能精准寻物
ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
- 通过生成未来视角图像,将导航转化为选最优视图的任务
- 在无地图环境下超越多数依赖地图的方法,实现顶尖性能
- 适合研究视觉语言模型与机器人自主导航的学者参考
视觉导航是自主家居助手机器人完成长期任务(如找物品)的核心能力。尽管近期方法利用大语言模型(LLMs)融入常识推理并提升探索效率,其规划仍受限于文本表示,难以捕捉空间占用和场景几何——这些对导航决策至关重要。本文探索视觉语言模型(VLMs)能否仅通过机载RGB/RGB-D流实现无地图视觉导航,释放其空间感知与规划潜力。提出ImagineNav++框架,通过想象力驱动导航:先由未来视角生成模块提炼人类导航偏好,生成语义有意义且探索潜力高的候选视角;再将这些想象视图作为视觉提示输入VLM,选择最信息量的视角。为保持空间一致性,设计选择性中央凹记忆机制,通过稀疏到密集的框架分层融合关键帧观测,构建紧凑而全面的记忆用于长期空间推理。该方法将目标导向导航分解为一系列可处理的点目标导航任务。在开放词汇物体与实例导航基准上的大量实验表明,ImagineNav++在无地图设置下达到当前最优性能,甚至超越多数基于地图的方法,凸显了场景想象与记忆在VLM-based空间推理中的重要性。
原文摘要 · Abstract (English)
Visual navigation is a fundamental capability for autonomous home-assistance robots, enabling long-horizon tasks such as object search. While recent methods have leveraged Large Language Models (LLMs) to incorporate commonsense reasoning and improve exploration efficiency, their planning remains constrained by textual representations, which cannot adequately capture spatial occupancy or scene geometry--critical factors for navigation decisions. We explore whether Vision-Language Models (VLMs) can achieve mapless visual navigation using only onboard RGB/RGB-D streams, unlocking their potential for spatial perception and planning. We achieve this through an imagination-powered navigation framework, ImagineNav++, which imagines future observation images from candidate robot views and translates navigation planning into a simple best-view image selection problem for VLMs. First, a future-view imagination module distills human navigation preferences to generate semantically meaningful viewpoints with high exploration potential. These imagined views then serve as visual prompts for the VLM to identify the most informative viewpoint. To maintain spatial consistency, we develop a selective foveation memory mechanism, which hierarchically integrates keyframe observations via a sparse-to-dense framework, constructing a compact yet comprehensive memory for long-term spatial reasoning. This approach transforms goal-oriented navigation into a series of tractable point-goal navigation tasks. Extensive experiments on open-vocabulary object and instance navigation benchmarks show that ImagineNav++ achieves SOTA performance in mapless settings, even surpassing most map-based methods, highlighting the importance of scene imagination and memory in VLM-based spatial reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。