用生成模型做通用实例分割，零样本泛化能力强。
gen2seg: Generative Models Enable Generalizable Instance Segmentation
- 用实例着色损失微调生成模型，实现无类别依赖分割。
- 在未见物体类型上表现接近SAM，细结构分割更优。
- 无需大规模预训练，生成模型自带跨类分组能力。
通过在扰动输入上预训练生成一致图像，生成模型自然学习到物体边界和场景构成。如何重用这些生成表示实现通用感知组织？我们仅在室内家具和汽车两类物体上，使用实例着色损失对Stable Diffusion和MAE（编码器+解码器）进行微调。令人惊讶的是，模型展现出强大的零样本泛化能力，能准确分割微调中未见过的物体类型和风格。这一现象甚至出现在仅在无标签ImageNet-1K上预训练的MAE上。在评估未见物体类型和风格时，最佳模型性能接近全监督的SAM，且在细结构与模糊边界分割上表现更优。相比之下，现有可提示分割架构或判别式预训练模型无法泛化。这表明生成模型内在具备可迁移的分组机制，即使缺乏互联网规模预训练也可跨类别、跨域转移。
原文摘要 · Abstract (English)
By pretraining to synthesize coherent images from perturbed inputs, generative models inherently learn to understand object boundaries and scene compositions. How can we repurpose these generative representations for general-purpose perceptual organization? We finetune Stable Diffusion and MAE (encoder+decoder) for category-agnostic instance segmentation using our instance coloring loss exclusively on a narrow set of object types (indoor furnishings and cars). Surprisingly, our models exhibit strong zero-shot generalization, accurately segmenting objects of types and styles unseen in finetuning. This holds even for MAE, which is pretrained on unlabeled ImageNet-1K only. When evaluated on unseen object types and styles, our best-performing models closely approach the heavily supervised SAM, and outperform it when segmenting fine structures and ambiguous boundaries. In contrast, existing promptable segmentation architectures or discriminatively pretrained models fail to generalize. This suggests that generative models learn an inherent grouping mechanism that transfers across categories and domains, even without internet-scale pretraining. Please see our website for additional qualitative figures, code, and a demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。