arXiv:2409.08156cs.CV2024-09被引 2

用参考图风格化人像,保留细节同时融合色彩纹理。

MagicStyle: Portrait Stylization Based on Reference Image

  • 分两阶段:先逆向提取内容与风格图像的注意力特征,再融合生成。
  • 在人像风格迁移中保持面部细节,颜色纹理融合自然。
  • 适合需要高质量人像艺术化处理的设计师或创作者。

扩散模型的发展显著推进了图像风格化研究,尤其是基于参考图像对内容图像进行风格迁移的任务,吸引了众多学者关注。该任务的主要挑战在于如何在保留内容图像细节的同时融入参考图像的颜色与纹理特征,这一挑战在处理具有复杂纹理的人像时尤为突出。为此,本文提出一种专为人像风格迁移设计的扩散模型方法——MagicStyle,包含两个阶段:内容与风格DDIM反演(CSDI)和特征融合前向(FFF)。CSDI阶段通过分别对内容图像和风格图像执行反向去噪过程,存储其自注意力机制中的查询、键和值特征;FFF阶段则通过精心设计的特征融合注意力机制(FFA),在前向去噪过程中将预存的特征信息和谐地融入扩散生成过程。我们进行了全面的对比实验与消融实验,验证了MagicStyle及FFA的有效性。

原文摘要 · Abstract (English)

The development of diffusion models has significantly advanced the research on image stylization, particularly in the area of stylizing a content image based on a given style image, which has attracted many scholars. The main challenge in this reference image stylization task lies in how to maintain the details of the content image while incorporating the color and texture features of the style image. This challenge becomes even more pronounced when the content image is a portrait which has complex textural details. To address this challenge, we propose a diffusion model-based reference image stylization method specifically for portraits, called MagicStyle. MagicStyle consists of two phases: Content and Style DDIM Inversion (CSDI) and Feature Fusion Forward (FFF). The CSDI phase involves a reverse denoising process, where DDIM Inversion is performed separately on the content image and the style image, storing the self-attention query, key and value features of both images during the inversion process. The FFF phase executes forward denoising, harmoniously integrating the texture and color information from the pre-stored feature queries, keys and values into the diffusion generation process based on our Well-designed Feature Fusion Attention (FFA). We conducted comprehensive comparative and ablation experiments to validate the effectiveness of our proposed MagicStyle and FFA.

人像风格化扩散模型特征融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。