提出解耦式交互抠图框架,支持多类型输入和多轮交互,提升多人场景抠图效果。
DFIMat: Decoupled Flexible Interactive Matting in Multi-Person Scenarios
- 将抠图拆分为定位与精修两阶段,结合语义理解与灵活用户输入。
- 在SMPMat数据集上达到94.3%的F-measure,显著优于现有方法。
- 适合需要高效精准抠图的设计师、视频编辑等专业用户。
交互式人像抠图旨在通过用户输入从图像中提取符合其意图的柔边人像。现有方法在复杂场景下表现不佳,主要源于三方面:(1) 多数采用紧密耦合网络,缺乏可解释性且建模能力弱;(2) 仅支持单一类型用户输入,影响意图理解与操作效率;(3) 忽视多轮交互特性。为此,我们提出DFIMat,一种解耦式灵活交互抠图框架。首先将任务解耦为两个子任务:基于场景语义与灵活输入的目标实例定位,以及实例级抠图精修。解耦设计使子任务更易学习,多类型输入进一步提升效果与效率。同时引入对比推理模块,增强跨轮次精修能力。针对多人抠图数据稀缺问题,构建了新型合成数据生成流程,生成更真实样本,并建立新数据集SMPMat。实验验证了DFIMat的显著优势,同时分析了不同输入类型的作用,为用户提供了实用指导。代码与数据集见https://github.com/JiaoSiyi/DFIMat。
原文摘要 · Abstract (English)
Interactive portrait matting refers to extracting the soft portrait from a given image that best meets the user's intent through their inputs. Existing methods often underperform in complex scenarios, mainly due to three factors. (1) Most works apply a tightly coupled network that directly predicts matting results, lacking interpretability and resulting in inadequate modeling. (2) Existing works are limited to a single type of user input, which is ineffective for intention understanding and also inefficient for user operation. (3) The multi-round characteristics have been under-explored, which is crucial for user interaction. To alleviate these limitations, we propose DFIMat, a decoupled framework that enables flexible interactive matting. Specifically, we first decouple the task into 2 sub-ones: localizing target instances by understanding scene semantics and the flexible user inputs, and conducting refinement for instance-level matting. We observe a clear performance gain from decoupling, as it makes sub-tasks easier to learn, and the flexible multi-type input further enhances both effectiveness and efficiency. DFIMat also considers the multi-round interaction property, where a contrastive reasoning module is designed to enhance cross-round refinement. Another limitation for multi-person matting task is the lack of training data. We address this by introducing a new synthetic data generation pipeline that can generate much more realistic samples than previous arts. A new large-scale dataset SMPMat is subsequently established. Experiments verify the significant superiority of DFIMat. With it, we also investigate the roles of different input types, providing valuable principles for users. Our code and dataset can be found at https://github.com/JiaoSiyi/DFIMat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。