让物体自然融入场景，基于可使用性原理实现精准插入。
Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
- 双流扩散模型同时优化图像与插入掩码，显式建模物体放置位置。
- 在300万数据上训练，对任意物体和场景都表现优异。
- 适合需要真实感图像合成的设计师与研究人员。
图像合成是常见图像编辑操作，涉及将前景物体融合到背景场景中。本文将‘可使用性’（Affordance）概念从以人为中心的图像合成扩展至更通用的物体-场景融合框架，解决前景物体与背景场景间的复杂交互问题。根据可使用性原则，定义了可使用性感知的物体插入任务，旨在通过多种位置提示，无缝将任意物体插入任意场景。为应对数据稀缺问题，构建了包含超过300万样本、覆盖3000多个物体类别的SAM-FB数据集。提出掩码感知双扩散（MADD）模型，采用双流架构，同时去噪RGB图像和插入掩码。通过在扩散过程中显式建模插入掩码，有效支持可使用性概念。大量实验表明，该方法优于当前最先进方法，在真实世界图像上也表现出强泛化能力。
原文摘要 · Abstract (English)
As a common image editing operation, image composition involves integrating foreground objects into background scenes. In this paper, we expand the application of the concept of Affordance from human-centered image composition tasks to a more general object-scene composition framework, addressing the complex interplay between foreground objects and background scenes. Following the principle of Affordance, we define the affordance-aware object insertion task, which aims to seamlessly insert any object into any scene with various position prompts. To address the limited data issue and incorporate this task, we constructed the SAM-FB dataset, which contains over 3 million examples across more than 3,000 object categories. Furthermore, we propose the Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream architecture to simultaneously denoise the RGB image and the insertion mask. By explicitly modeling the insertion mask in the diffusion process, MADD effectively facilitates the notion of affordance. Extensive experimental results show that our method outperforms the state-of-the-art methods and exhibits strong generalization performance on in-the-wild images. Please refer to our code on https://github.com/KaKituken/affordance-aware-any.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。