让一张图的风格迁移到多物体复杂场景，无需调参就能精准还原细节。
Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild
- 通过增强自注意力机制实现跨图像外观匹配，自动迁移局部风格。
- 在多个物体复杂场景中保持语义对齐与外观真实度，性能领先。
- 适合需要快速复现特定场景风格的研究者和设计师使用。
基于示例的语义图像合成在保持语义内容的同时，保留示例图像的外观特征。传统结构引导模型（如ControlNet）仅依赖文本提示控制外观，无法直接使用示例图像作为输入。近期无微调方法通过预训练扩散模型中的增强自注意力机制实现隐式跨图像匹配来传递局部外观，但通常局限于单对象或前景对象的外观迁移，在包含多个对象的复杂场景中表现不佳。为此，我们提出AM-Adapter（外观匹配适配器），用于解决真实世界中多对象的示例驱动图像合成问题，支持从一张场景级图像中迁移多对象外观。AM-Adapter能自动将场景级输入的局部外观迁移至目标图像，并可选地让用户定义物体细节映射到生成图像中的特定位置。其可学习框架通过整合分割图的语义信息，增强增强自注意力中的跨图像匹配能力。为解耦生成与匹配过程，采用分阶段训练：先训练结构引导与生成网络，再冻结其余部分，仅训练匹配适配器。推理时引入自动化示例检索方法，高效选择示例图像-分割对。尽管仅使用极少可学习参数，AM-Adapter仍达到当前最优性能，显著提升语义对齐与局部外观保真度。大量消融实验验证了设计合理性。代码与权重将公开。
原文摘要 · Abstract (English)
Exemplar-based semantic image synthesis generates images aligned with semantic content while preserving the appearance of an exemplar. Conventional structure-guidance models like ControlNet, are limited as they rely solely on text prompts to control appearance and cannot utilize exemplar images as input. Recent tuning-free approaches address this by transferring local appearance via implicit cross-image matching in the augmented self-attention mechanism of pre-trained diffusion models. However, prior works are often restricted to single-object cases or foreground object appearance transfer, struggling with complex scenes involving multiple objects. To overcome this, we propose AM-Adapter (Appearance Matching Adapter) to address exemplar-based semantic image synthesis in-the-wild, enabling multi-object appearance transfer from a single scene-level image. AM-Adapter automatically transfers local appearances from the scene-level input. AM-Adapter alternatively provides controllability to map user-defined object details to specific locations in the synthesized images. Our learnable framework enhances cross-image matching within augmented self-attention by integrating semantic information from segmentation maps. To disentangle generation and matching, we adopt stage-wise training. We first train the structure-guidance and generation networks, followed by training the matching adapter while keeping the others frozen. During inference, we introduce an automated exemplar retrieval method for selecting exemplar image-segmentation pairs efficiently. Despite utilizing minimal learnable parameters, AM-Adapter achieves state-of-the-art performance, excelling in both semantic alignment and local appearance fidelity. Extensive ablations validate our design choices. Code and weights will be released.: https://cvlab-kaist.github.io/AM-Adapter/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。