用检测模型自动找物体放置位置,效果远超现有方法。
BOOTPLACE: Bootstrapped Object Placement with Detection Transformers
- 将物体放置转为检测问题,通过补全背景训练专用检测器。
- 在Cityscapes和OPA数据集上IoU显著提升,优于当前最优方法。
- 适合需要精准图像合成的视觉生成与编辑任务。
本文针对图像到图像的复制粘贴合成问题,聚焦于物体放置学习。以往方法依赖生成模型减少密集标注依赖,但常限制其对复杂数据分布的建模能力;另一些基于稀疏对比损失的变换器网络则因正则化过松导致放置不精确。我们提出BOOTPLACE,将物体放置重构为‘按检测放置’问题:首先在减去目标物体的背景上训练专用检测变换器,引入多物体监督以识别合适放置区域;随后根据互补特征语义关联目标物体与检测区域。通过在随机减去物体的图像上应用自举训练策略,结合大规模配对数据增强,强制模型实现有意义的放置。在标准基准测试中,该方法在城市景观(Cityscapes)和开放图像合成(OPA)数据集上均显著优于现有最先进方法,尤其在交并比(IoU)指标上表现突出。消融实验进一步验证了方法的组合性与泛化能力,用户评估也支持其有效性。
原文摘要 · Abstract (English)
In this paper, we tackle the copy-paste image-to-image composition problem with a focus on object placement learning. Prior methods have leveraged generative models to reduce the reliance for dense supervision. However, this often limits their capacity to model complex data distributions. Alternatively, transformer networks with a sparse contrastive loss have been explored, but their over-relaxed regularization often leads to imprecise object placement. We introduce BOOTPLACE, a novel paradigm that formulates object placement as a placement-by-detection problem. Our approach begins by identifying suitable regions of interest for object placement. This is achieved by training a specialized detection transformer on object-subtracted backgrounds, enhanced with multi-object supervisions. It then semantically associates each target compositing object with detected regions based on their complementary characteristics. Through a boostrapped training approach applied to randomly object-subtracted images, our model enforces meaningful placements through extensive paired data augmentation. Experimental results on established benchmarks demonstrate BOOTPLACE's superior performance in object repositioning, markedly surpassing state-of-the-art baselines on Cityscapes and OPA datasets with notable improvements in IOU scores. Additional ablation studies further showcase the compositionality and generalizability of our approach, supported by user study evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。