arXiv:2411.08196cs.CV2024-11被引 3

发现扩散模型隐空间可解耦,无需训练即可精准零样本语义编辑

Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing

  • 通过文本提示直接提取编辑方向,实现无训练语义控制
  • 实验验证隐空间具备解耦性,特定属性可独立调控
  • 适用于需要精细图像编辑的研究者与开发者

扩散 Transformer(DiT)在文本引导图像生成中表现卓越。本文研究其隐空间特性,发现两个关键性质:第一,DiT 隐空间天然具有语义解耦性,不同语义属性可通过特定编辑方向控制;第二,精确编辑需使用完整的联合隐空间,仅图像或文本编码均信息不足。我们证明编辑方向可直接从文本提示中获得,实现无需额外训练或掩码标注的精准语义控制。基于此提出 Encode-Identify-Manipulate(EIM)框架:先编码源图与文本提示得到联合隐向量,再用提出的海森评分蒸馏采样(HSDS)方法识别目标属性的编辑方向,结合文本提示操纵隐向量。此外,提出新度量指标量化扩散模型隐空间的解耦程度。在自建基准数据集上的大量实验与分析验证了 DiT 的解耦特性及 EIM 框架的有效性。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) have recently achieved remarkable success in text-guided image generation. In image editing, DiTs project text and image inputs to a joint latent space, from which they decode and synthesize new images. However, it remains largely unexplored how multimodal information collectively forms this joint space and how they guide the semantics of the synthesized images. In this paper, we investigate the latent space of DiT models and uncover two key properties: First, DiT's latent space is inherently semantically disentangled, where different semantic attributes can be controlled by specific editing directions. Second, consistent semantic editing requires utilizing the entire joint latent space, as neither encoded image nor text alone contains enough semantic information. We show that these editing directions can be obtained directly from text prompts, enabling precise semantic control without additional training or mask annotations. Based on these insights, we propose a simple yet effective Encode-Identify-Manipulate (EIM) framework for zero-shot fine-grained image editing. Specifically, we first encode both the given source image and the text prompt that describes the image, to obtain the joint latent embedding. Then, using our proposed Hessian Score Distillation Sampling (HSDS) method, we identify editing directions that control specific target attributes while preserving other image features. These directions are guided by text prompts and used to manipulate the latent embeddings. Moreover, we propose a new metric to quantify the disentanglement degree of the latent space of diffusion models. Extensive experiment results on our new curated benchmark dataset and analysis demonstrate DiT's disentanglement properties and effectiveness of the EIM framework.

图像编辑扩散模型隐空间解耦零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。