arXiv:2606.01621cs.CVcs.RO2026-06被引 2

用像素点导航,让视觉语言模型少算多走。

Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation

论文配图:Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigation
图 1 · 摘自论文原文
  • 把导航任务转为预测可见可走的像素点,直接生成3D目标点。
  • 在R2R-CE上仅需7.75次模型调用,成功率54.1%,比之前少6倍。
  • 适合追求高效推理、减少模型调用次数的研究者和应用落地。

视觉语言模型(VLM)已成为连续环境视觉语言导航(VLN-CE)的通用基础。然而,多数基于VLM的方法将导航建模为低层动作预测,该接口存在歧义、依赖短视运动原语,且因重复调用VLM而效率低下。本文提出Goal2Pixel,一种纯像素级导航范式,将VLN-CE重定义为可导航像素定位任务。模型不预测动作,而是直接在图像平面上预测一个可见的可行走像素,并将其反投影为3D导航目标点。对于非前向动作,通过在图像平面附加辅助指令区域实现:左侧区域表示左转,右侧表示右转,底部表示停止。为支持长时序导航,引入可见性感知的关键帧记忆机制,实现紧凑且信息丰富的历史表征。为适配预训练VLM完成像素定位任务,设计语义嵌入与坐标感知辅助损失。实验表明,Goal2Pixel在多项指标上达到当前最优表现,且显著减少VLM调用次数:在R2R-CE Val-Unseen上,仅需7.75次调用/每回合,成功率(SR)达54.1%,路径相似度(SPL)为52.5%,相较直接动作预测方法(46.62次调用,SR 32.9%)减少6倍。该趋势同样在RxR-CE数据集上成立。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have become a common foundation for vision-and-language navigation in continuous environments (VLN-CE). Yet most VLM-based methods cast navigation as low-level action prediction, an interface that is ambiguous, tied to short-horizon motion primitives, and inefficient due to repeated VLM querying. We propose Goal2Pixel, a pure pixel-based paradigm that reformulates VLN-CE as navigable pixel grounding. Rather than predicting actions, Goal2Pixel uses the image plane as a unified spatial interface between VLM reasoning and robot motion: the model predicts a visible navigable pixel to the agent, which is back-projected into a 3D waypoint for forward navigation. For non-forward actions, we append auxiliary directive regions to the image plane, where the left/right/bottom regions are interpreted as turning left, turning right, and stopping, respectively. To enable long-horizon navigation, we propose a visibility-aware keyframe memory for compact and informative history representation. To adapt pretrained VLMs to navigable pixel grounding, we introduce semantic embeddings and coordinate-aware auxiliary losses. Goal2Pixel achieves competitive state-of-the-art performance while requiring fewer VLM inference calls than prior methods. On R2R-CE Val-Unseen it achieves 54.1% SR and 52.5% SPL with just 7.75 VLM calls per episode, 6x fewer than the 46.62 required by direct action prediction at 32.9% SR. The same trend holds on RxR-CE.Project Page: https://baobao0926.github.io/Goal2Pixel/.

视觉导航像素定位高效推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。