arXiv:2604.07916cs.CV2026-04

无需训练即可精准分割任意语言描述的图像区域。

Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation

论文配图:Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation
图 1 · 摘自论文原文
  • 通过推理引导提示生成与自修正机制,提升分割鲁棒性。
  • 在显式与隐式表达上均超越现有方法,开放世界场景表现优异。
  • 适合需要快速部署、不依赖标注数据的视觉语言应用。

指代表达分割(RES)旨在根据自然语言描述定位图像区域,是连接视觉与语言理解的关键任务。现有方法严重依赖大规模标注数据,且仅适用于显式或隐式表达,难以泛化到任意指代表达。尽管最近的 Segment Anything Model 3(SAM3)在可提示概念分割中表现出色,但其应用于 RES 仍面临挑战:(1) 对长或隐含表达处理能力弱;(2) 简单耦合多模态大语言模型(MLLM)会使结果过度依赖 MLLM 推理,无法优化 SAM3 的分割输出。为此,我们提出 Tarot-SAM3,一种无需训练的新型框架,可实现对任意指代表达的准确分割。该框架包含两个关键阶段:第一阶段为表达推理解析器(ERI),引入推理辅助提示选项,支持结构化表达解析与评估感知重述,将任意查询转化为鲁棒的异构提示,以生成可靠的 SAM3 分割掩码;第二阶段为掩码自修正(MSR),在不同提示类型中选择最优掩码,并利用 DINOv3 提供的丰富特征关系,对比 ERI 输出中的判别区域,推断目标区域归属,从而修正过分割与欠分割问题。大量实验表明,Tarot-SAM3 在显式与隐式表达基准及开放世界场景下均取得优异性能。消融实验证实了各阶段的有效性。

原文摘要 · Abstract (English)

Referring Expression Segmentation (RES) aims to segment image regions described by natural-language expressions, serving as a bridge between vision and language understanding. Existing RES methods, however, rely heavily on large annotated datasets and are limited to either explicit or implicit expressions, hindering their ability to generalize to any referring expression. Recently, the Segment Anything Model 3 (SAM3) has shown impressive robustness in Promptable Concept Segmentation. Nonetheless, applying it to RES remains challenging: (1) SAM3 struggles with longer or implicit expressions; (2) naive coupling of SAM3 with a multimodal large language model (MLLM) makes the final results overly dependent on the MLLM's reasoning capability, without enabling refinement of SAM3's segmentation outputs. To this end, we present Tarot-SAM3, a novel training-free framework that can accurately segment from any referring expression. Specifically, Tarot-SAM3 consists of two key phases. First, the Expression Reasoning Interpreter (ERI) phase introduces reasoning-assisted prompt options to support structured expression parsing and evaluation-aware rephrasing. This transforms arbitrary queries into robust heterogeneous prompts for generating reliable masks with SAM3. Second, the Mask Self-Refining (MSR) phase selects the best mask across prompt types and performs self-refinement by leveraging rich feature relationships from DINOv3 to compare discriminative regions among ERI outputs. It then infers region affiliation to the target, thereby correcting over- and under-segmentation. Extensive experiments demonstrate that Tarot-SAM3 achieves strong performance on both explicit and implicit RES benchmarks, as well as open-world scenarios. Ablation studies further validate the effectiveness of each phase.

指代分割SAM3零样本自修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。