无需训练即可通过U-Net跳连实现风格与内容分离迁移
Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion
- 利用U-Net第三编码器块的跳连传递空间信息
- 实现文本编辑、精准修改与风格迁移，内容对齐更优
- 无需训练，适合快速图像编辑场景
扩散模型在图像生成中的进展促使研究者深入分析U-Net架构中各组件在图像编辑中的作用。以往研究集中于瓶颈层（h-space）、交叉注意力、自注意力和解码层，但对U-Net跳连的整体作用尚未明确。我们系统分析发现，第三编码器块的残差连接承载了重建图像的主要空间信息，将内容与风格分离：内容信息经此路径传递，风格信息则由反向解码路径携带。基于此，我们提出方法SkipInject，仅通过注入该块特征即可实现文本驱动编辑、精确修改和风格迁移。与当前主流方法对比，我们的方法在内容对齐与结构保持之间取得最优平衡。
原文摘要 · Abstract (English)
Recent advances in diffusion models for image generation have led to detailed examinations of several components within the U-Net architecture for image editing. While previous studies have focused on the bottleneck layer (h-space), cross-attention, self-attention, and decoding layers, the overall role of the skip connections of the U-Net itself has not been specifically addressed. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style, passed by the remaining stream in the opposed decoding layer. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our method, SkipInject, to state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。