无需训练,用四个视觉线索精准选点,显著提升图像分割效果。
PinPoint: Prompting with Informative Interior Points

- 融合四个视觉线索生成共识图,选出避开边界的精准点
- 在五个点的预算下,cIoU提升12-18个百分点
- 无需微调或强化学习,仅需两次VLM调用,适合快速部署
现代指代图像分割系统通常将视觉语言模型(VLM)用于定位,并结合可提示的分割器(如SAM)生成掩码。现有无训练方案普遍落后于微调或强化学习优化的专用模型,但其差距源于VLM定位、SAM能力还是提示设计尚不明确。本文表明,差距主要来自提示模糊性:VLM提供的边界框使SAM难以判断框内哪些像素属于目标对象。内部点是自然的消歧工具,但其位置至关重要;以往方法使用随机采样的点,常落在边界、干扰物或背景上,甚至劣于仅用边界框。监督与强化学习方法通过训练VLM预测更好点来弥补差距,但我们证明该训练并非必要。在相同五点预算下,改用稳定且信息量高的点选择策略,可在RefCOCO/+/g数据集上实现12-18个百分点的累计交并比(cIoU)提升,且所有模型保持固定。基于此,我们提出PinPoint——一种确定性的、无需训练的点选择器,融合四种视觉线索生成共识图,选取紧凑且空间分布多样、远离边界的点,并利用冻结的VLM对每个点打标。无需任何任务特定训练,PinPoint在相同架构上达到监督与强化学习优化模型的性能,每查询仅需两次VLM调用。
原文摘要 · Abstract (English)
Modern referring image segmentation pipelines couple a vision-language model (VLM) for grounding with a promptable segmenter such as the Segment Anything Model (SAM) for mask generation. Prior training-free instances of this recipe consistently trail fine-tuned and reinforcement-learning (RL)-tuned specialists, and it has been unclear whether the gap comes from the VLM's grounding, SAM's capacity, or the prompt. We show that the gap is dominated by prompt ambiguity: a VLM-proposed bounding box (bbox) leaves SAM to guess which pixels inside the bbox belong to the object the expression denotes. Interior points are the natural disambiguator, but where they fall matters; prior work relies on naively sampled points that land on boundaries, distractors, and background clutter, and can even hurt performance compared to the bbox alone. Supervised and RL-tuned methods close this gap by training a VLM to predict better points; we show that this training is unnecessary. At a matched budget of five interior points, replacing naive sampling with stable, informative point selection improves cumulative Intersection-over-Union (cIoU) by 12-18 points across RefCOCO/+/g, with every model fixed. We turn this observation into PinPoint, a deterministic, training-free point selector that fuses four visual cues into a consensus map, selects compact, spatially diverse points away from boundaries, and uses the frozen VLM to label each point. Without any task-specific training, PinPoint matches supervised and RL-tuned specialists on the same stack while issuing only two VLM calls per query.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。