无需微调的图像编辑方法,兼顾细节保真与文本对齐。
Tuning-Free Image Editing with Fidelity and Editability via Unified Latent Diffusion Model
- 通过自注意力与交叉注意力约束统一优化保真度与可编辑性。
- 动态调度器缓解梯度冲突,避免过度或不足编辑。
- 适合需要稳定图像编辑效果的研究者与开发者使用。
在文本驱动图像编辑中,保真度与可编辑性的平衡至关重要,现有方法常因结构保持依赖注意力注入、可编辑性依赖预训练文生图模型的文本对齐能力,缺乏显式统一机制。本文提出UnifyEdit,一种无需微调的方法,通过扩散潜空间优化,在统一框架内实现保真与可编辑性的平衡。设计自注意力(SA)保真约束与交叉注意力(CA)对齐约束,但二者同时应用易引发梯度冲突,导致过编辑或欠编辑。为此,引入自适应时间步调度器,动态调节约束影响,引导潜空间向最优平衡点收敛。大量定量与定性实验验证其有效性,在多种编辑任务中优于当前最先进方法,显著提升结构保持与文本对齐的鲁棒性。代码将开源于 https://github.com/CUC-MIPG/UnifyEdit。
原文摘要 · Abstract (English)
Balancing fidelity and editability is essential in text-based image editing (TIE), where failures commonly lead to over- or under-editing issues. Existing methods typically rely on attention injections for structure preservation and leverage the inherent text alignment capabilities of pre-trained text-to-image (T2I) models for editability, but they lack explicit and unified mechanisms to properly balance these two objectives. In this work, we introduce UnifyEdit, a tuning-free method that performs diffusion latent optimization to enable a balanced integration of fidelity and editability within a unified framework. Unlike direct attention injections, we develop two attention-based constraints: a self-attention (SA) preservation constraint for structural fidelity, and a cross-attention (CA) alignment constraint to enhance text alignment for improved editability. However, simultaneously applying both constraints can lead to gradient conflicts, where the dominance of one constraint results in over- or under-editing. To address this challenge, we introduce an adaptive time-step scheduler that dynamically adjusts the influence of these constraints, guiding the diffusion latent toward an optimal balance. Extensive quantitative and qualitative experiments validate the effectiveness of our approach, demonstrating its superiority in achieving a robust balance between structure preservation and text alignment across various editing tasks, outperforming other state-of-the-art methods. The source code will be available at https://github.com/CUC-MIPG/UnifyEdit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。