arXiv:2606.04046cs.CVcs.AI2026-06中稿 · ICML

通过分步生成注意力计划,解决视觉语言模型的误判问题。

Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation

论文配图:Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
图 1 · 摘自论文原文
  • 先构建场景图再逐步分解任务,实现由粗到精的注意力规划。
  • 在多个基准上显著减少视觉幻觉,误识别率降低超过40%。
  • 适合需要精准感知的机器人操作与导航任务,兼顾效率与准确。

在具身视觉-语言决策任务(如机器人操作与导航)中,视觉-语言模型(VLMs)和视觉-语言-动作模型(VLAs)各有优势:前者擅长长期规划,后者擅长反应式控制。然而两者均受制于相同的感知瓶颈——因无法区分任务相关物体与干扰物,导致视觉幻觉。理论上,准确识别关键物体并聚焦其上是突破该限制的关键。直接的一步式聚焦无效,因有效聚焦需深层场景理解。为此,我们提出SceneDiver,一种利用VLM长程规划能力的粗到精注意力计划生成方法:先构建整体场景图建立初步理解,再通过识别、理解、分析的迭代循环逐步分解任务。为支持反应式控制,还设计轻量适配器,将这种有意识的聚焦能力蒸馏至VLAs。在标准具身AI基准上的评估表明,该方法显著降低了VLMs与VLAs的视觉幻觉,同时保持快速执行任务的计算效率。代码与数据已公开:https://future-item.github.io/SceneDiver。

原文摘要 · Abstract (English)

In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models (VLMs & VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual hallucinations arise due to the models' inability to distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation. A straightforward solution is one-step focus: directly attending to essential objects. However, this approach proves ineffective because effective focus inherently requires deep scene understanding. To this end, we propose SceneDiver, a coarse-to-fine focus plan generation method for VLMs leveraging their long-term planning abilities, that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through an iterative cycle of recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter for distilling the deliberate focus ability into VLAs. Evaluations on standard embodied AI benchmarks confirm that our method substantially reduces visual hallucinations for both VLMs and VLAs, while preserving computational efficiency in tasks requiring fast execution. Our code and data are released at: https://future-item.github.io/SceneDiver.

具身智能视觉推理注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。