用视觉提示分步引导机器人执行复杂指令,提升精准度与泛化能力。
VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
- 通过视觉提示将任务分解为规划与执行两阶段,避免直接映射的局限
- 在仿真和真实场景中均超越现有端到端模型,在未见场景下表现更优
- 适合需要高精度控制与强泛化能力的具身智能任务
视觉-语言-动作(VLA)模型通常将视觉观测与语言指令直接映射为控制信号,这种‘黑箱’方式要求单次前向传播同时完成指令理解、空间定位与底层控制,常导致空间精度不足且在分布外场景下鲁棒性差。为此,我们提出VP-VLA,一种基于结构化视觉提示接口的双系统框架,解耦高层推理与底层执行。具体而言,‘系统2规划器’将复杂指令拆分为子任务,并识别目标物体与终点位置,这些空间锚点以十字线、边界框等模态一致的视觉提示形式直接渲染在原生RGB观测空间中,避免了密集掩码、属性图或专用控制表示带来的模态错配。在训练中引入新型辅助视觉定位目标,增强提示感知能力。随后,‘系统1控制器’在提示引导下生成精确的低层执行动作。大量仿真与真实世界实验表明,VP-VLA显著优于现有端到端基线模型,包括QwenOFT与GR00T-N1.6。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are rendered directly within the native RGB observation space as modality-consistent visual prompts, such as crosshairs and bounding boxes. This avoids the modality mismatch introduced by dense masks, affordance maps, or additional control-specific representations. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Extensive experiments in simulation and real world demonstrate that VP-VLA surpasses state-of-the-art end-to-end baselines including QwenOFT and GR00T-N1.6. Project page: https://visualprompt-vla.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。