arXiv:2411.00626cs.CV2024-11ICCV被引 16

零样本图像抠图模型ZIM,实现任意物体高精度抠图。

ZIM: Zero-Shot Image Matting for Anything

  • 用标签转换器构建无标注的SA1B-Matte数据集,训练SAM生成精细抠图。
  • 提出分层解码器与提示感知注意力机制,提升细节表现力。
  • 在微级抠图测试集上超越现有方法,适用于修复与3D重建等任务。

最近的分割基础模型Segment Anything Model(SAM)具备强大的零样本分割能力,但在生成细粒度精确掩码方面表现不足。为此,我们提出一种新型零样本图像抠图模型ZIM,主要贡献有二:其一,设计标签转换器将分割标签转化为详细抠图标签,构建无需人工标注的新数据集SA1B-Matte;在该数据集上训练SAM后,仍保持零样本能力并生成高精度抠图。其二,设计包含分层像素解码器的零样本抠图模型,增强掩码表示能力,并引入提示感知掩码注意力机制,使模型能聚焦于视觉提示指定区域。我们在新提出的MicroMat-3K测试集上评估ZIM,该数据集包含高质量微级别抠图标签。实验结果表明,ZIM在细粒度掩码生成和零样本泛化方面均优于现有方法。此外,我们展示了ZIM在图像修复、3D NeRF等需高精度掩码的下游任务中的广泛适用性。本工作为推进零样本抠图及其应用提供了坚实基础。代码已开源:https://github.com/naver-ai/ZIM。

原文摘要 · Abstract (English)

The recent segmentation foundation model, Segment Anything Model (SAM), exhibits strong zero-shot segmentation capabilities, but it falls short in generating fine-grained precise masks. To address this limitation, we propose a novel zero-shot image matting model, called ZIM, with two key contributions: First, we develop a label converter that transforms segmentation labels into detailed matte labels, constructing the new SA1B-Matte dataset without costly manual annotations. Training SAM with this dataset enables it to generate precise matte masks while maintaining its zero-shot capability. Second, we design the zero-shot matting model equipped with a hierarchical pixel decoder to enhance mask representation, along with a prompt-aware masked attention mechanism to improve performance by enabling the model to focus on regions specified by visual prompts. We evaluate ZIM using the newly introduced MicroMat-3K test set, which contains high-quality micro-level matte labels. Experimental results show that ZIM outperforms existing methods in fine-grained mask generation and zero-shot generalization. Furthermore, we demonstrate the versatility of ZIM in various downstream tasks requiring precise masks, such as image inpainting and 3D NeRF. Our contributions provide a robust foundation for advancing zero-shot matting and its downstream applications across a wide range of computer vision tasks. The code is available at https://github.com/naver-ai/ZIM.

图像抠图零样本分割模型3D重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。