arXiv:2508.04655cs.CVcs.AI2025-08AAAI被引 30

X-SAM让大模型能精准分割任意物体,支持交互式视觉提示。

X-SAM: From Segment Anything to Any Segmentation

  • 构建统一框架,融合语言与视觉感知,实现像素级理解。
  • 在多类分割任务上超越现有方法,支持多掩码和类别特定分割。
  • 适合需要高精度图像分割的AI研究者与开发者使用。

大型语言模型(LLMs)具备广泛的知识表征能力,但在像素级感知理解方面存在天然不足。尽管分割一切模型(SAM)在视觉提示驱动的图像分割上取得显著进展,但在多掩码预测和类别特定分割任务中仍存在明显局限,且无法将所有分割任务整合于统一模型架构中。为此,我们提出X-SAM,一种精简的多模态大语言模型(MLLM)框架,将分割范式从“分割一切”扩展至“任意分割”。具体而言,我们引入一个新颖的统一框架,增强MLLM对像素级感知的理解能力。此外,我们提出一项新任务——视觉定位分割(Visual GrounDed, VGD),通过交互式视觉提示分割所有实例对象,赋予MLLM视觉锚定、像素级可解释能力。为支持跨多种数据源的有效训练,我们设计了一种统一训练策略,支持多数据集协同训练。实验结果表明,X-SAM在多个图像分割基准上达到领先性能,凸显其在多模态像素级视觉理解方面的高效性。代码已开源:https://github.com/wanghao9610/X-SAM。

原文摘要 · Abstract (English)

Large Language Models (LLMs) demonstrate strong capabilities in broad knowledge representation, yet they are inherently deficient in pixel-level perceptual understanding. Although the Segment Anything Model (SAM) represents a significant advancement in visual-prompt-driven image segmentation, it exhibits notable limitations in multi-mask prediction and category-specific segmentation tasks, and it cannot integrate all segmentation tasks within a unified model architecture. To address these limitations, we present X-SAM, a streamlined Multimodal Large Language Model (MLLM) framework that extends the segmentation paradigm from \textit{segment anything} to \textit{any segmentation}. Specifically, we introduce a novel unified framework that enables more advanced pixel-level perceptual comprehension for MLLMs. Furthermore, we propose a new segmentation task, termed Visual GrounDed (VGD) segmentation, which segments all instance objects with interactive visual prompts and empowers MLLMs with visual grounded, pixel-wise interpretative capabilities. To enable effective training on diverse data sources, we present a unified training strategy that supports co-training across multiple datasets. Experimental results demonstrate that X-SAM achieves state-of-the-art performance on a wide range of image segmentation benchmarks, highlighting its efficiency for multimodal, pixel-level visual understanding. Code is available at https://github.com/wanghao9610/X-SAM.

图像分割多模态视觉理解大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。