不依赖训练数据,用文本桥梁生成符合特定风格的图像。
Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics
- 用视觉语言模型提取图像内容,再通过ChatGPT融合风格描述
- 在无训练情况下实现跨风格语义一致的图像生成
- 适合需要快速试多种艺术风格的设计师或创意工作者
传统上,风格主要被视为色彩、笔触和光照等艺术元素。然而,同一语义主体(如人物、船只、房屋)在不同艺术传统中差异显著,表明风格还包含底层语义。为此,本文提出一种零样本图像风格化生成方案,将图像到图像问题转化为图像到文本再到图像的过程。首先利用视觉语言模型(如BLIP)生成描述输入图像内容的文本,包括物体及其位置;随后,通过ChatGPT将风格关键词扩展为详细风格描述,并与内容文本融合;最后使用扩散模型根据融合后的文本生成图像。为增强模型对多种风格的适应性,提出一种微调策略,在交叉注意力中注入文本和风格约束,确保输出图像在目标风格下保持高语义一致性。为验证性能,构建了一个涵盖多种风格与场景的基准数据集,并引入两项新指标。尽管方法简单,该方案在零样本条件下仍生成高度逼真的图像,尤其在保持高保真语义方面表现优异。
原文摘要 · Abstract (English)
Traditionally, style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. However, identical semantic subjects, like people, boats, and houses, can vary significantly across different artistic traditions, indicating that style also encompasses the underlying semantics. Therefore, in this study, we propose a zero-shot scheme for image variation with coordinated semantics. Specifically, our scheme transforms the image-to-image problem into an image-to-text-to-image problem. The image-to-text operation employs vision-language models e.g., BLIP) to generate text describing the content of the input image, including the objects and their positions. Subsequently, the input style keyword is elaborated into a detailed description of this style and then merged with the content text using the reasoning capabilities of ChatGPT. Finally, the text-to-image operation utilizes a Diffusion model to generate images based on the text prompt. To enable the Diffusion model to accommodate more styles, we propose a fine-tuning strategy that injects text and style constraints into cross-attention. This ensures that the output image exhibits similar semantics in the desired style. To validate the performance of the proposed scheme, we constructed a benchmark comprising images of various styles and scenes and introduced two novel metrics. Despite its simplicity, our scheme yields highly plausible results in a zero-shot manner, particularly for generating stylized images with high-fidelity semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。