分离视觉推理与定位,让模型精准找图中目标。
PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seeking

- 推理与感知分开:先决定看什么,再由专用工具找位置。
- 用掩码搜索和广度优先策略,减少重复错误区域的探索。
- 新基准测试验证模型定位能力,适合研究视觉推理的学者。
本文研究多轮视觉推理,发现当前多模态大模型在定位目标时反复失败,导致轨迹冗长且重复。原因在于推理与感知耦合在一个模型中:模型同时进行推理和定位,定位不准引发更多推理轮次,造成轨迹膨胀。为此,我们提出PixelEyes,一种显式解耦推理与感知的多轮视觉推理智能体——推理器决定要找什么,专用感知工具回答在哪里。具体包括:1)掩码引导的视觉搜索,通过指代分割模型实现精确定位,使推理器无需补偿定位误差;2)语义区域广度优先搜索(BFS),避免因反复裁剪错误区域造成的冗余循环。为内化这些能力,我们构建了PixelEyes-6K数据集,通过重构已有数据中的专家轨迹生成训练样本,显式嵌入掩码搜索与BFS逻辑。此外,我们提出Pinpoint-Bench,一个无提示的视觉搜索基准,问题中不提供位置线索,使用实例级掩码与边界框分离定位失败与推理失败,支持对注意力盲视等失效模式的细粒度分析。当前最先进多模态模型在此基准上仍有巨大提升空间,证明其挑战性与价值。代码与模型已开源。
原文摘要 · Abstract (English)
This paper explores multi-turn visual reasoning and observes that MLLMs repeatedly fail to localize the target, leading to long, redundant trajectories. We attribute this failure to the entanglement of reasoning and perception within a single model, the MLLM reasons and localizes simultaneously, and inaccurate localization triggers additional reasoning turns that bloat the trajectory. To solve this problem, we propose PixelEyes, a multi-turn visual reasoning agent that explicitly decouples reasoning from perception, i.e., the reasoner decides what to look for, while a specialized perception tool answers where it is. Specifically, PixelEyes introduces 1) Mask-guided Visual Search. A referring segmentation model is invoked to provide mask-precise localization, freeing the reasoner from the need to compensate for imprecise grounding. 2) Semantic-region Breadth-first Search (BFS). To eliminate redundant loops caused by repeatedly cropping incorrect sub-regions, we organize exploration as a breadth-first search over semantic regions. To internalize these capabilities, we construct the PixelEyes-6K dataset by resynthesizing expert trajectories from existing data. This explicitly embeds our mask-guided search and BFS logic into the model. We further introduce Pinpoint-Bench, a zero-hint visual search benchmark, i.e., no location cues are provided in the question, with instance-level masks and bounding boxes that separate localization failures from reasoning failures, enabling fine-grained analysis of failure modes such as inattentional blindness. Recent state-of-the-art MLLMs and visual reasoning agents leave large headroom on Pinpoint-Bench, demonstrating its quality and difficulty. Code and models are open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。