无需训练,通过调整风格分布实现零样本图像风格迁移
Z-STAR+: A Zero-shot Style Transfer Method via Adjusting Style Distribution
- 利用扩散模型隐空间中的自然风格与内容分布,直接提取风格信息
- 双去噪路径分离内容与风格,交叉注意力重加权提升风格匹配度
- 适合追求高质量、无训练风格迁移的视觉生成研究者
风格迁移面临的核心挑战在于如何有效表征风格。传统方法依赖于基于二阶统计或对比学习的风格损失来约束生成结果中的风格表示,但这类预定义风格表示常限制风格表达并引入伪影。本文发现,原始扩散模型的隐层特征天然包含自然的风格与内容分布,可直接提取风格信息,并无缝融合生成先验至内容图像,无需重新训练。提出Z-STAR+方法,采用双去噪路径在隐空间中分别表征内容与风格参考,再以风格隐码引导内容图像的去噪过程。引入交叉注意力重加权模块,利用局部内容特征查询最适配输入块的风格信息,使生成结果的风格分布与风格图像一致。同时设计缩放自适应实例归一化,缓解风格与生成图像间全局色彩分布不一致问题。通过理论分析与大量实验验证,证明了该方法在零样本风格迁移中的有效性与优越性。
原文摘要 · Abstract (English)
Style transfer presents a significant challenge, primarily centered on identifying an appropriate style representation. Conventional methods employ style loss, derived from second-order statistics or contrastive learning, to constrain style representation in the stylized result. However, these pre-defined style representations often limit stylistic expression, leading to artifacts. In contrast to existing approaches, we have discovered that latent features in vanilla diffusion models inherently contain natural style and content distributions. This allows for direct extraction of style information and seamless integration of generative priors into the content image without necessitating retraining. Our method adopts dual denoising paths to represent content and style references in latent space, subsequently guiding the content image denoising process with style latent codes. We introduce a Cross-attention Reweighting module that utilizes local content features to query style image information best suited to the input patch, thereby aligning the style distribution of the stylized results with that of the style image. Furthermore, we design a scaled adaptive instance normalization to mitigate inconsistencies in color distribution between style and stylized images on a global scale. Through theoretical analysis and extensive experimentation, we demonstrate the effectiveness and superiority of our diffusion-based \uline{z}ero-shot \uline{s}tyle \uline{t}ransfer via \uline{a}djusting style dist\uline{r}ibution, termed Z-STAR+.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。