通过提取图像差异并精准注入,实现跨模型的图像类比生成。
Difference Inversion: Interpolate and Isolate the Difference with Token Consistency for Image Analogy Generation
- 用差值插值和标记一致性损失精确分离源图像差异
- 在多个扩散模型上生成结果优于现有方法,定量与定性均更优
- 适用于Stable Diffusion等通用模型,无需特定指令格式
给定输入图像A、A'和B,如何生成满足A:A'::B:B'关系的图像B'?现有方法如视觉上下文学习或视觉指令通常依赖特定模型(如InstructPix2Pix、修复模型),难以推广至通用扩散模型(如Stable Diffusion、SDXL),易引入偏见或限制编辑能力。本文提出Difference Inversion,通过提取A与A'之间的差异,并将其精准应用于B以生成合理的B'。为摆脱模型依赖,采用适配于稳定扩散模型的“全提示”结构,而非“指令提示”。为此,提出:1)差值插值(Delta Interpolation)识别差异;2)标记一致性损失确保训练精度;3)标记嵌入零初始化避免偏差。大量实验表明,该方法在定量与定性指标上均优于现有基线,在模型无关性下生成更合理的B'。
原文摘要 · Abstract (English)
How can we generate an image B' that satisfies A:A'::B:B', given the input images A,A' and B? Recent works have tackled this challenge through approaches like visual in-context learning or visual instruction. However, these methods are typically limited to specific models (e.g. InstructPix2Pix. Inpainting models) rather than general diffusion models (e.g. Stable Diffusion, SDXL). This dependency may lead to inherited biases or lower editing capabilities. In this paper, we propose Difference Inversion, a method that isolates only the difference from A and A' and applies it to B to generate a plausible B'. To address model dependency, it is crucial to structure prompts in the form of a "Full Prompt" suitable for input to stable diffusion models, rather than using an "Instruction Prompt". To this end, we accurately extract the Difference between A and A' and combine it with the prompt of B, enabling a plug-and-play application of the difference. To extract a precise difference, we first identify it through 1) Delta Interpolation. Additionally, to ensure accurate training, we propose the 2) Token Consistency Loss and 3) Zero Initialization of Token Embeddings. Our extensive experiments demonstrate that Difference Inversion outperforms existing baselines both quantitatively and qualitatively, indicating its ability to generate more feasible B' in a model-agnostic manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。