用视觉语义判别机制提升指代表达分割的定位准确性。
DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentation

- 先生成候选区域,再让大模型做两阶段筛选,避免坐标预测偏差。
- 在三个数据集上达到领先性能,显著减少误检和漏检。
- 适合需要高精度目标定位的视觉语言任务研究者使用。
指代表达分割(RES)旨在为语言描述中的对象生成像素级掩码。现有基于多模态大模型(MLLM)的方法通常依赖单次坐标预测进行视觉定位,将连续空间位置离散化为文本标记,易引发定位偏差和对齐误差。为此,我们提出DRAgent,一种由MLLM驱动的判别式推理(DR)框架用于RES。DRAgent不依赖MLLM直接生成定位坐标,而是首先构建由检测器生成的候选区域空间,随后利用MLLM作为视觉-语义目标判别器。具体地,MLLM通过两阶段判别机制实现可靠的目标选择:先筛选高召回候选,再进行实例级验证。选定的目标框随后作为空间提示输入基础分割模型,生成最终像素级掩码。此外,我们构建了自一致性过滤的推理链数据管道,用于基于LoRA的微调,提供更可靠的监督信号以增强MLLM的判别推理能力。实验表明,DRAgent在RefCOCO、RefCOCO+和RefCOCOg上均取得具有竞争力的性能。
原文摘要 · Abstract (English)
Referring Expression Segmentation (RES) aims to generate a pixel-level mask for the object specified by a language expression. Recent methods based on multimodal large language models (MLLMs) often rely on one-pass coordinate prediction for visual localization, which serializes continuous spatial locations as discrete text tokens and may lead to localization bias and alignment errors. To address these issues, we propose DRAgent, an MLLM-driven discriminative reasoning (DR) framework for RES. Instead of requiring the MLLM to generate localization coordinates, DRAgent first constructs a detector-generated candidate space and then uses the MLLM as a visual-semantic target discriminator. Specifically, the MLLM performs reliable target selection among potential distractors through a two-stage DR mechanism, which first screens high-recall candidates and then performs instance-wise verification. The selected target box is subsequently used as a spatial prompt for a foundation segmentation model to produce the final pixel-level mask. Furthermore, we construct a self-consistency-filtered reasoning-chain data pipeline for LoRA-based fine-tuning, providing more reliable supervision for enhancing the MLLM's discriminative reasoning capability. Experiments demonstrate that DRAgent achieves competitive performance on RefCOCO, RefCOCO+, and RefCOCOg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。