arXiv:2510.10160cs.CVcs.AI2025-10NeurIPS被引 3

模仿人类看图思路,用Mamba模型提升复杂描述的图像分割能力

SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation

  • 分两阶段处理:先全局理解,再细节聚焦,模拟人类视觉认知
  • 在新基准上显著优于现有方法,尤其擅长处理模糊或含干扰项的描述
  • 适合需要理解复杂语言指令的图像分割场景

指代图像分割(RIS)旨在根据自然语言表达分割图像中的目标对象。尽管近期方法借助预训练视觉主干网络和更大训练数据集取得了优异结果,但主要关注简单表达——如“红色汽车”或“左边的女孩”这类简短清晰的名词短语。这种简化使RIS退化为关键词匹配问题,限制了模型处理语言指代歧义的能力。本文识别出两个现实挑战场景:含干扰物体的表达(多实体+上下文线索)和类别隐含的表达(未明确说明对象类别)。为此,提出SaFiRe框架,模拟人类双阶段认知过程:先形成整体理解,再通过细节检视逐步优化。该设计契合Mamba的扫描-更新特性,支持高效多轮精炼,计算复杂度线性增长。同时构建新基准aRefCOCO,用于评估模型在模糊指代表达下的表现。在标准与新数据集上的大量实验表明,SaFiRe显著优于当前最优基线。

原文摘要 · Abstract (English)

Referring Image Segmentation (RIS) aims to segment the target object in an image given a natural language expression. While recent methods leverage pre-trained vision backbones and more training corpus to achieve impressive results, they predominantly focus on simple expressions--short, clear noun phrases like "red car" or "left girl". This simplification often reduces RIS to a key word/concept matching problem, limiting the model's ability to handle referential ambiguity in expressions. In this work, we identify two challenging real-world scenarios: object-distracting expressions, which involve multiple entities with contextual cues, and category-implicit expressions, where the object class is not explicitly stated. To address the challenges, we propose a novel framework, SaFiRe, which mimics the human two-phase cognitive process--first forming a global understanding, then refining it through detail-oriented inspection. This is naturally supported by Mamba's scan-then-update property, which aligns with our phased design and enables efficient multi-cycle refinement with linear complexity. We further introduce aRefCOCO, a new benchmark designed to evaluate RIS models under ambiguous referring expressions. Extensive experiments on both standard and proposed datasets demonstrate the superiority of SaFiRe over state-of-the-art baselines.

图像分割语言理解Mamba指代消解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。