无需文字提示,实现开放世界图像分割,性能超越现有方法。
WOW-Seg: A Word-free Open World Segmentation Model

- 用视觉提示模块将掩码转为视觉令牌,对齐特征空间。
- 通过级联注意力抑制实例间干扰,提升分割精度。
- 在7662类数据集上验证,参数量仅为1/8仍达顶尖水平。
开放世界图像分割旨在应对真实世界中无限多样的物体类别,实现精准分割与语义理解。传统封闭集分割方法难以适应复杂开放场景,而像SAM这样的基础分割模型虽有强分割能力,但语义理解较弱。为此,我们提出无文字开放世界分割模型WOW-Seg,通过引入新的视觉提示模块Mask2Token,将图像掩码转化为视觉令牌并对其与视觉语言大模型(VLLM)特征空间对齐;同时设计级联注意力掩码机制,解耦不同实例间信息,有效缓解实例间干扰。我们还构建了首个大规模开放世界区域识别基准:区域识别数据集RR-7K,包含7,662个类别,是目前最丰富的类别密集型区域识别数据集。WOW-Seg在LVIS数据集上取得89.7的语义相似度和82.4的语义交并比,优于先前最优方法,且仅需其1/8的参数量,充分证明其卓越的开放世界泛化能力。代码与资源见https://github.com/AAwcAA/WOW-Seg-Meta。
原文摘要 · Abstract (English)
Open world image segmentation aims to achieve precise segmentation and semantic understanding of targets within images by addressing the infinitely open set of object categories encountered in the real world. However, traditional closed-set segmentation approaches struggle to adapt to complex open world scenarios, while foundation segmentation models such as SAM exhibit notable discrepancies between their strong segmentation capabilities and relatively weaker semantic understanding. To bridge these discrepancies, we propose WOW-Seg, a Word-free Open World Segmentation model for segmenting and recognizing objects from open-set categories. Specifically, WOW-Seg introduces a novel visual prompt module, Mask2Token, which transforms image masks into visual tokens and ensures their alignment with the VLLM feature space. Moreover, we introduce the Cascade Attention Mask to decouple information across different instances. This approach mitigates inter-instance interference, leading to a significant improvement in model performance. We further construct an open world region recognition test benchmark: the Region Recognition Dataset (RR-7K). With 7,662 classes, it represents the most extensive category-rich region recognition dataset to date. WOW-Seg attains strong results on the LVIS dataset, achieving a semantic similarity of 89.7 and a semantic IoU of 82.4. This performance surpasses the previous SOTA while using only one-eighth the parameter count. These results underscore the strong open world generalization capabilities of WOW-Seg. The code and related resources are available at https://github.com/AAwcAA/WOW-Seg-Meta.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。