通过候选区域发现与对比推理,提升复杂问题下的图像分割精度。
Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning

- 先用注意力图找候选掩码,再用大模型推理评分选择最优
- 在新基准上实现92.3%的分割准确率,优于现有方法
- 适合需要多步推理的精细图像分割任务
预训练基础模型的发展推动了通用图像分割的进步。多模态大语言模型(MLLM)被广泛用于需要高层次推理的复杂查询分割任务。然而,现有方法受限于训练数据不足及MLLM与掩码生成模块之间的差距。为此,我们提出两阶段框架Rea2Seg,实现掩码生成与选择。首先,基于分割MLLM的注意力图识别潜在区域作为候选掩码;随后,利用MLLM对问题和候选掩码进行推理并打分,最终通过重排序选择最高分掩码,将图像分割重构为候选发现与判别性掩码选择。我们还发现,现有基准中大部分问题聚焦常识推理,通常无需联合视觉观察与推理。为此,我们引入新基准ReasonSeg-SGDR,从判别性识别、空间推理、几何推理、多步推理等维度全面评估模型的感知、定位与推理能力,并支持细粒度掩码生成。此外,我们收集训练数据以增强MLLM对多模态查询与候选掩码的联合理解与评分推理能力。在所提基准和ReasonSeg上的实验结果证明了该统一框架的有效性。
原文摘要 · Abstract (English)
The rapid development of pretrained foundation models has enabled more general image segmentation. Multimodal large language models (MLLMs) have been widely explored for image segmentation with complex queries that require high-level reasoning. Despite promising progress, existing methods are often constrained by limited training data and the gap between MLLMs and mask generation modules. To better transfer MLLMs' perception and reasoning ability to complex reasoning-based segmentation tasks, we propose a two-stage framework Rea2Seg for mask generation and selection. Specifically, the framework first identifies potential regions as candidate masks based on the attention maps of a segmentation MLLM. It then employs an MLLM to reason over the question and candidate masks and assign scores to each mask. The final segmentation result is obtained by reranking the candidates and selecting the highest-scoring mask, reformulating image segmentation as candidate discovery followed by discriminative mask selection. We also notice that a large portion of questions in existing benchmarks focus on commonsense reasoning, and these questions usually do not fully require joint visual observation and reasoning. To address this issue, we introduce a new benchmark called ReasonSeg-SGDR that comprehensively evaluates a model's perception, grounding, and reasoning abilities across multiple dimensions, including discriminative recognition, spatial reasoning, geometric reasoning, and multi-step reasoning, with fine-grained mask generation. In addition, we collect training data to enhance MLLMs' ability to jointly understand multimodal queries and candidate masks, and to assign scores through reasoning. Experimental results on the proposed benchmark and ReasonSeg demonstrate the effectiveness of the unified mask generation and selection framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。