arXiv:2507.01738cs.CV2025-07ICCV被引 16

拆解视觉与认知模块,提升图像指代分割的精准与泛化能力

DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergy

  • 将指代分割拆分为感知与认知两模块,系统分析性能瓶颈
  • 引入回环协同机制,显著提升图文理解与分割精度
  • 无需修改结构即可适应单/多指代场景,适合通用部署

指代图像分割(RIS)旨在根据自然语言表达定位图像中的目标对象。现有方法多聚焦于优化视觉-语言交互与细粒度定位,但对当前RIS框架根本瓶颈的系统性分析仍不足。为此,我们提出DeRIS,将RIS分解为感知与认知两个核心组件,实现对主要性能瓶颈的系统分析。研究发现,当前模型的主要限制并非感知能力不足,而是多模态认知能力有限。为此,我们设计回环协同机制,增强感知与认知模块间的协同,实现精确分割的同时提升图像-文本理解鲁棒性。此外,我们引入一种简单的非指代样本转换数据增强方法,缓解目标存在性判断中长尾分布问题。值得注意的是,DeRIS在无需特殊结构调整的情况下,天然适用于单指代与多指代场景,显著提升通用性。代码与模型已开源:https://github.com/Dmmm1997/DeRIS。

原文摘要 · Abstract (English)

Referring Image Segmentation (RIS) is a challenging task that aims to segment objects in an image based on natural language expressions. While prior studies have predominantly concentrated on improving vision-language interactions and achieving fine-grained localization, a systematic analysis of the fundamental bottlenecks in existing RIS frameworks remains underexplored. To bridge this gap, we propose DeRIS, a novel framework that decomposes RIS into two key components: perception and cognition. This modular decomposition facilitates a systematic analysis of the primary bottlenecks impeding RIS performance. Our findings reveal that the predominant limitation lies not in perceptual deficiencies, but in the insufficient multi-modal cognitive capacity of current models. To mitigate this, we propose a Loopback Synergy mechanism, which enhances the synergy between the perception and cognition modules, thereby enabling precise segmentation while simultaneously improving robust image-text comprehension. Additionally, we analyze and introduce a simple non-referent sample conversion data augmentation to address the long-tail distribution issue related to target existence judgement in general scenarios. Notably, DeRIS demonstrates inherent adaptability to both non- and multi-referents scenarios without requiring specialized architectural modifications, enhancing its general applicability. The codes and models are available at https://github.com/Dmmm1997/DeRIS.

图像分割多模态指代理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。