用单张图精准控制生成风格，不干扰文字提示
StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models
- 通过风格标记符将图像风格对齐文本空间，实现无损控制
- 在Style30k数据集上训练，从单图提取准确风格特征
- 适合需要精细风格迁移的生成任务，如设计、艺术创作
尽管文本到图像生成中涌现出大量控制扩散过程的方法，但有效控制图像风格仍是难题。许多基于适配器的方法在去噪过程中引入图像表示条件，但这些条件与词嵌入空间不一致，导致图像与文本控制相互干扰，可能损失文本语义信息。为此，需解决两大挑战：如何注入风格表示而不影响文本控制效果；如何仅凭单张参考图像获取准确风格特征。为此，我们提出StyleTokenizer，一种零样本风格控制生成方法，通过风格标记符将风格表示与文本表示对齐，显著降低对文本提示的干扰。此外，我们构建了名为Style30k的标注良好的风格数据集，用于训练风格特征提取器，可精准捕捉风格信息并排除内容干扰。实验表明，该方法能充分保留参考图像的风格特征，生成既符合目标风格又契合文本提示的高质量图像。代码与数据集已开源。
原文摘要 · Abstract (English)
Despite the burst of innovative methods for controlling the diffusion process, effectively controlling image styles in text-to-image generation remains a challenging task. Many adapter-based methods impose image representation conditions on the denoising process to accomplish image control. However these conditions are not aligned with the word embedding space, leading to interference between image and text control conditions and the potential loss of semantic information from the text prompt. Addressing this issue involves two key challenges. Firstly, how to inject the style representation without compromising the effectiveness of text representation in control. Secondly, how to obtain the accurate style representation from a single reference image. To tackle these challenges, we introduce StyleTokenizer, a zero-shot style control image generation method that aligns style representation with text representation using a style tokenizer. This alignment effectively minimizes the impact on the effectiveness of text prompts. Furthermore, we collect a well-labeled style dataset named Style30k to train a style feature extractor capable of accurately representing style while excluding other content information. Experimental results demonstrate that our method fully grasps the style characteristics of the reference image, generating appealing images that are consistent with both the target image style and text prompt. The code and dataset are available at https://github.com/alipay/style-tokenizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。