arXiv:2603.10703cs.CVcs.CY2026-03中稿 · CVPR被引 2

让导航模型看清路况并精准分割障碍物,生成有深度的无障碍指引。

WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation

  • 统一语言与分割的架构,通过多尺度查询投影实现像素级定位。
  • 在41,000张行人视角图像上实现高精度语义分割与相对深度估计。
  • 适合无障碍导航、智能助行系统等真实场景应用。

实现可访问的行人导航需同时理解复杂城市场景的语义与空间信息,而现有大视觉语言模型(LVLMs)因缺乏显式定位,常产生物体幻觉和不可靠的深度推理,限制其在无障碍引导中的应用。我们提出WalkGPT,一种用于新型“接地导航指引”任务的像素级接地LVLM,将语言推理与分割统一于单一架构中,实现深度感知的无障碍引导。给定行人视角图像与导航问题,WalkGPT生成包含可通行与危险区域分割掩码及相对深度估计的对话响应。模型引入多尺度查询投影器(MSQP),通过跨空间层级聚合图像与文本令牌;以及校准文本投影器(CTP),在区域对齐损失指导下将语言嵌入映射为分割感知表示。该设计无需用户输入提示或锚点,实现细粒度定位与深度推断,生成完整且真实的导航指引。我们还构建了包含41,000张行人视角图像的PAVE大规模基准数据集,配以无障碍相关问题与深度接地答案。实验表明,WalkGPT在接地推理与分割性能上均表现优异。代码与数据集已公开于项目网站。

原文摘要 · Abstract (English)

Ensuring accessible pedestrian navigation requires reasoning about both semantic and spatial aspects of complex urban scenes, a challenge that existing Large Vision-Language Models (LVLMs) struggle to meet. Although these models can describe visual content, their lack of explicit grounding leads to object hallucinations and unreliable depth reasoning, limiting their usefulness for accessibility guidance. We introduce WalkGPT, a pixel-grounded LVLM for the new task of Grounded Navigation Guide, unifying language reasoning and segmentation within a single architecture for depth-aware accessibility guidance. Given a pedestrian-view image and a navigation query, WalkGPT generates a conversational response with segmentation masks that delineate accessible and harmful features, along with relative depth estimation. The model incorporates a Multi-Scale Query Projector (MSQP) that shapes the final image tokens by aggregating them along text tokens across spatial hierarchies, and a Calibrated Text Projector (CTP), guided by a proposed Region Alignment Loss, that maps language embeddings into segmentation-aware representations. These components enable fine-grained grounding and depth inference without user-provided cues or anchor points, allowing the model to generate complete and realistic navigation guidance. We also introduce PAVE, a large-scale benchmark of 41k pedestrian-view images paired with accessibility-aware questions and depth-grounded answers. Experiments show that WalkGPT achieves strong grounded reasoning and segmentation performance. The source code and dataset are available on the \href{https://sites.google.com/view/walkgpt-26/home}{project website}.

导航生成视觉语言模型分割无障碍

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。