分离几何与外观,实现物体精准拼接且不失细节。
Geometry-Editable and Appearance-Preserving Object Compositon
- 用语义嵌入隐式控制几何变化,再通过跨注意力对齐外观特征。
- 在公开数据集上实现高精度几何编辑与外观一致性保留。
- 适合需要精细图像合成与可控编辑的视觉设计场景。
通用物体合成(GOC)旨在将目标物体无缝融入背景场景,同时实现所需的几何调整并保留其细粒度外观细节。现有方法依赖语义嵌入结合先进扩散模型实现可编辑几何生成,但这些紧凑嵌入仅编码高层语义,不可避免丢失细节。本文提出解耦式几何可编辑与外观保持扩散模型(DGAD),首先利用语义嵌入隐式捕捉目标几何变换,再通过跨注意力检索机制将细粒度外观特征与几何编辑后的表示对齐,从而实现精确几何编辑与忠实外观保留。DGAD基于CLIP/DINO及参考网络提取语义嵌入与外观保持表征,并以解耦方式集成至编码-解码流程。先将语义嵌入注入具备强空间推理能力的预训练扩散模型,隐式学习对象几何,实现灵活操作与有效可编辑性;随后设计密集跨注意力机制,利用隐式学习的几何信息检索并空间对齐外观特征,确保外观一致性。大量实验在公开基准上验证了所提DGAD框架的有效性。
原文摘要 · Abstract (English)
General object composition (GOC) aims to seamlessly integrate a target object into a background scene with desired geometric properties, while simultaneously preserving its fine-grained appearance details. Recent approaches derive semantic embeddings and integrate them into advanced diffusion models to enable geometry-editable generation. However, these highly compact embeddings encode only high-level semantic cues and inevitably discard fine-grained appearance details. We introduce a Disentangled Geometry-editable and Appearance-preserving Diffusion (DGAD) model that first leverages semantic embeddings to implicitly capture the desired geometric transformations and then employs a cross-attention retrieval mechanism to align fine-grained appearance features with the geometry-edited representation, facilitating both precise geometry editing and faithful appearance preservation in object composition. Specifically, DGAD builds on CLIP/DINO-derived and reference networks to extract semantic embeddings and appearance-preserving representations, which are then seamlessly integrated into the encoding and decoding pipelines in a disentangled manner. We first integrate the semantic embeddings into pre-trained diffusion models that exhibit strong spatial reasoning capabilities to implicitly capture object geometry, thereby facilitating flexible object manipulation and ensuring effective editability. Then, we design a dense cross-attention mechanism that leverages the implicitly learned object geometry to retrieve and spatially align appearance features with their corresponding regions, ensuring faithful appearance consistency. Extensive experiments on public benchmarks demonstrate the effectiveness of the proposed DGAD framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。