用对比学习自动修正指令图像编辑数据,提升生成结果与指令的一致性。
Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning

- 通过对比学习捕捉图像前后语义变化,自动修正指令-图像对
- 修复后数据集含12万+样本,训练模型更准确遵循指令
- 适用于扩散模型,可在任意扩散步骤保持指令一致性
自然语言指令为图像编辑提供了直观的控制方式,但深度学习模型常因高质量训练数据稀缺而表现不佳。以往方法依赖文本到图像生成模型合成原始与编辑图像对,但生成结果常与指令不匹配,影响训练效果。为此,本文提出Instruct-CLIP(I-CLIP),一种自监督方法,通过对比学习识别原始与编辑图像间的语义差异,自动优化现有数据集中的指令对齐。同时,该方法可处理噪声潜在空间图像和扩散时间步,适用于训练潜在扩散模型(LDMs),并在扩散过程任意阶段高效确保指令与图像变化一致。我们使用I-CLIP修正InstructPix2Pix数据集,获得超过12万条精修样本,并基于新型基于I-CLIP的损失函数微调模型。结果模型生成的编辑内容与指令更吻合。代码与数据集已开源。
原文摘要 · Abstract (English)
Although natural language instructions offer an intuitive way to guide automated image editing, deep-learning models often struggle to achieve high-quality results, largely due to the difficulty of creating large, high-quality training datasets. To do this, previous approaches have typically relied on text-to-image (T2I) generative models to produce pairs of original and edited images that simulate the input/output of an instruction-guided image-editing model. However, these image pairs often fail to align with the specified edit instructions due to the limitations of T2I models, which negatively impacts models trained on such datasets. To address this, we present Instruct-CLIP (I-CLIP), a selfsupervised method that learns the semantic changes between original and edited images to refine and better align the instructions in existing datasets. Furthermore, we adapt Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it can be used to train latent diffusion models (LDMs) and efficiently enforce alignment between the edit instruction and the image changes in latent space at any step of the diffusion pipeline. We use Instruct-CLIP to correct the InstructPix2Pix dataset and get over 120K refined samples we then use to fine-tune their model, guided by our novel I-CLIP-based loss function. The resulting model can produce edits that are more aligned with the given instructions. Our code and dataset are available at https://github.com/SherryXTChen/Instruct-CLIP.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。