arXiv:2410.06593cs.CV2024-10被引 4

用真实场景数据训练图像抠图模型,提升复杂场景下的精度。

Towards Natural Image Matting in the Wild via Real-Scenario Prior

  • 构建了38251张真实复杂场景的人体抠图数据集COCO-Matting
  • 提出SEMat架构,融合边缘与透明度特征,实现高精度抠图
  • 适合需要真实场景抠图的视觉应用开发者使用

现有方法尝试将强大的交互式分割模型(如SAM)用于交互式抠图,并基于合成抠图数据集进行微调。然而,合成数据训练的模型在复杂和遮挡场景下泛化能力差。为此,本文基于COCO数据集构建了新的抠图数据集COCO-Matting,通过附属融合与掩码转抠图策略,从真实复杂图像中提取人体实例级α抠图,共包含38,251张图像。此外,现有基于SAM的抠图方法仅冻结SAM并训练轻量解码器,未能充分发挥预训练模型潜力。因此,本文提出SEMat,重构网络结构与训练目标:特征对齐变换器学习细粒度边缘与透明度特征;抠图对齐解码器将粗掩码转换为高精度抠图;正则化与三图损失保留预训练先验,并促使掩码解码器输出含三图语义信息的抠图结果。在七个不同数据集上的实验验证了方法的优越性,证明其在真实场景交互式图像抠图中的有效性。代码、模型与数据集已开源。

原文摘要 · Abstract (English)

Recent approaches attempt to adapt powerful interactive segmentation models, such as SAM, to interactive matting and fine-tune the models based on synthetic matting datasets. However, models trained on synthetic data fail to generalize to complex and occlusion scenes. We address this challenge by proposing a new matting dataset based on the COCO dataset, namely COCO-Matting. Specifically, the construction of our COCO-Matting includes accessory fusion and mask-to-matte, which selects real-world complex images from COCO and converts semantic segmentation masks to matting labels. The built COCO-Matting comprises an extensive collection of 38,251 human instance-level alpha mattes in complex natural scenarios. Furthermore, existing SAM-based matting methods extract intermediate features and masks from a frozen SAM and only train a lightweight matting decoder by end-to-end matting losses, which do not fully exploit the potential of the pre-trained SAM. Thus, we propose SEMat which revamps the network architecture and training objectives. For network architecture, the proposed feature-aligned transformer learns to extract fine-grained edge and transparency features. The proposed matte-aligned decoder aims to segment matting-specific objects and convert coarse masks into high-precision mattes. For training objectives, the proposed regularization and trimap loss aim to retain the prior from the pre-trained model and push the matting logits extracted from the mask decoder to contain trimap-based semantic information. Extensive experiments across seven diverse datasets demonstrate the superior performance of our method, proving its efficacy in interactive natural image matting. We open-source our code, models, and dataset at https://github.com/XiaRho/SEMat.

图像抠图真实场景SAM数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。