用扩散模型提升抠图细节,支持视觉提示交互。
SDMatte: Grafting Diffusion Models for Interactive Matting
- 将文本交互转为视觉提示,利用扩散模型先验增强细节建模。
- 融合坐标与不透明度嵌入,提升对位置和边缘的敏感度。
- 设计掩码自注意力机制,聚焦提示区域,改善抠图精度。
近期交互式抠图方法在捕捉物体主体区域方面表现良好,但在边缘区域提取精细细节方面仍有不足。基于数十亿图像-文本对训练的扩散模型,展现出强大的复杂数据分布建模能力与逼真纹理生成能力,同时具备鲁棒的文本驱动交互能力,成为解决交互式抠图问题的有力候选。为此,我们提出SDMatte——一种基于扩散模型的交互式抠图方法,主要贡献有三:首先,利用扩散模型的强大先验,将文本驱动交互能力转化为视觉提示驱动交互能力,实现更自然的交互方式;其次,将视觉提示的坐标嵌入与目标对象的不透明度嵌入融入U-Net结构,增强模型对空间位置和不透明度信息的敏感性;第三,提出一种掩码自注意力机制,使模型能聚焦于视觉提示指定的区域,显著提升性能。在多个数据集上的大量实验验证了该方法的有效性。代码与模型已公开于https://github.com/vivoCameraResearch/SDMatte。
原文摘要 · Abstract (English)
Recent interactive matting methods have shown satisfactory performance in capturing the primary regions of objects, but they fall short in extracting fine-grained details in edge regions. Diffusion models trained on billions of image-text pairs, demonstrate exceptional capability in modeling highly complex data distributions and synthesizing realistic texture details, while exhibiting robust text-driven interaction capabilities, making them an attractive solution for interactive matting. To this end, we propose SDMatte, a diffusion-driven interactive matting model, with three key contributions. First, we exploit the powerful priors of diffusion models and transform the text-driven interaction capability into visual prompt-driven interaction capability to enable interactive matting. Second, we integrate coordinate embeddings of visual prompts and opacity embeddings of target objects into U-Net, enhancing SDMatte's sensitivity to spatial position information and opacity information. Third, we propose a masked self-attention mechanism that enables the model to focus on areas specified by visual prompts, leading to better performance. Extensive experiments on multiple datasets demonstrate the superior performance of our method, validating its effectiveness in interactive matting. Our code and model are available at https://github.com/vivoCameraResearch/SDMatte.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。