用自回归生成模型实现像素级分割，提升细节捕捉与推理速度
ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
- 将图像生成与分割结合，通过视觉令牌还原生成掩码
- 在多个数据集上超越现有方法，推理速度显著提升
- 适合需要高精度分割和快速响应的应用场景
我们提出一种基于自回归生成的图像分割新范式（ARGenSeg），在统一框架中实现多模态理解与像素级感知。以往将分割融入多模态大模型的方法通常依赖边界点或专用分割头，使用离散表示或语义提示输入任务特定解码器，限制了大模型对细粒度视觉细节的捕捉能力。为此，我们设计了一种基于图像生成的分割框架，可自然生成目标对象的密集掩码。利用多模态大模型输出视觉令牌，并通过通用VQ-VAE解码为图像，使分割完全依赖于大模型的像素级理解。为降低推理延迟，采用跨尺度预测策略并行生成所需视觉令牌。大量实验表明，该方法在多个分割数据集上优于现有最先进方法，同时实现显著的推理加速，且保持强大的理解能力。
原文摘要 · Abstract (English)
We propose a novel AutoRegressive Generation-based paradigm for image Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level perception within a unified framework. Prior works integrating image segmentation into multimodal large language models (MLLMs) typically employ either boundary points representation or dedicated segmentation heads. These methods rely on discrete representations or semantic prompts fed into task-specific decoders, which limits the ability of the MLLM to capture fine-grained visual details. To address these challenges, we introduce a segmentation framework for MLLM based on image generation, which naturally produces dense masks for target objects. We leverage MLLM to output visual tokens and detokenize them into images using an universal VQ-VAE, making the segmentation fully dependent on the pixel-level understanding of the MLLM. To reduce inference latency, we employ a next-scale-prediction strategy to generate required visual tokens in parallel. Extensive experiments demonstrate that our method surpasses prior state-of-the-art approaches on multiple segmentation datasets with a remarkable boost in inference speed, while maintaining strong understanding capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。