用自回归建模实现精准图像分割,突破传统方法局限。
Seg-VAR: Image Segmentation with Visual Autoregressive Modeling
- 将分割转为条件自回归掩码生成,通过潜在表示学习替代判别训练。
- 在COCO和Cityscapes上均超越现有生成与判别方法,尤其在实例分割上提升显著。
- 适合研究视觉自回归、空间感知模型的开发者与研究人员参考。
尽管视觉自回归建模(VAR)在图像生成中展现出潜力,但其在需要精细低层空间感知的分割任务中的应用仍待探索。受经典Mask2Former多尺度建模启发,本文提出Seg-VAR,将分割重新定义为条件自回归掩码生成问题,通过替换判别学习为潜在学习过程实现。方法包含三个核心组件:(1) 图像编码器从输入图像生成潜在先验;(2) 空间感知的seglat编码器,利用位置敏感的颜色映射将分割掩码转换为离散潜在标记以区分实例;(3) 解码器从这些潜在标记重建掩码。采用多阶段训练策略:首先通过图像-掩码联合训练学习seglat表示,再优化潜在变换,最后对齐图像编码器生成的潜在与seglat分布。实验表明,Seg-VAR在多种分割任务与验证基准上均优于以往判别与生成方法。通过将分割视为分层序列预测任务,为自回归推理融入空间感知视觉系统开辟新路径。代码将发布于https://github.com/rkzheng99/Seg-VAR。
原文摘要 · Abstract (English)
While visual autoregressive modeling (VAR) strategies have shed light on image generation with the autoregressive models, their potential for segmentation, a task that requires precise low-level spatial perception, remains unexplored. Inspired by the multi-scale modeling of classic Mask2Former-based models, we propose Seg-VAR, a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem. This is achieved by replacing the discriminative learning with the latent learning process. Specifically, our method incorporates three core components: (1) an image encoder generating latent priors from input images, (2) a spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens using a location-sensitive color mapping to distinguish instances, and (3) a decoder reconstructing masks from these latents. A multi-stage training strategy is introduced: first learning seglat representations via image-seglat joint training, then refining latent transformations, and finally aligning image-encoder-derived latents with seglat distributions. Experiments show Seg-VAR outperforms previous discriminative and generative methods on various segmentation tasks and validation benchmarks. By framing segmentation as a sequential hierarchical prediction task, Seg-VAR opens new avenues for integrating autoregressive reasoning into spatial-aware vision systems. Code will be available at https://github.com/rkzheng99/Seg-VAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。