arXiv:2412.14496cs.CV2024-12被引 1

用图文联合数据训练,让图像风格与内容彻底分离。

WikiStyle+: A Multimodal Approach to Content-Style Representation Disentanglement for Artistic Image Stylization

  • 构建图文对应的艺术作品数据集,支持多模态输入
  • 通过可学习跨注意力层实现内容与风格的精准分离
  • 适合需要高质量艺术化图像生成的研究者和开发者

艺术图像风格迁移旨在将文本或图像提供的内容以目标风格呈现,其核心在于内容与风格的有效解耦。然而,现有方法主要依赖图像监督,存在两大问题:1)模型仅支持单一模态输入(内容或风格);2)解耦不充分导致参考图像中的内容泄露。为此,本文提出一种多模态内容-风格解耦方法。我们构建了包含艺术作品及其对应文本描述的《WikiStyle+》数据集,用于风格与内容的多模态标注。基于该数据集,提出一种由解耦表征引导的扩散模型:首先使用Q-Formers学习解耦表示,再通过可学习的多步交叉注意力层将其注入预训练扩散模型。实验表明,该方法在多模态监督下实现了参考图像中内容与风格的彻底解耦,从而生成更贴合参考风格艺术特征的精细化风格化图像。代码将在论文录用后公开。

原文摘要 · Abstract (English)

Artistic image stylization aims to render the content provided by text or image with the target style, where content and style decoupling is the key to achieve satisfactory results. However, current methods for content and style disentanglement primarily rely on image supervision, which leads to two problems: 1) models can only support one modality for style or content input;2) incomplete disentanglement resulting in content leakage from the reference image. To address the above issues, this paper proposes a multimodal approach to content-style disentanglement for artistic image stylization. We construct a \textit{WikiStyle+} dataset consists of artworks with corresponding textual descriptions for style and content. Based on the multimodal dataset, we propose a disentangled representations-guided diffusion model. The disentangled representations are first learned by Q-Formers and then injected into a pre-trained diffusion model using learnable multi-step cross-attention layers. Experimental results show that our method achieves a thorough disentanglement of content and style in reference images under multimodal supervision, thereby enabling more refined stylization that aligns with the artistic characteristics of the reference style. The code of our method will be available upon acceptance.

图像风格迁移多模态扩散模型解耦表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。