给扩散模型注入组合创造力,无需重训就能生成创意混合物。
Redefining <Creative> in Dictionary: Towards an Enhanced Semantic Understanding of Creative Generation
- 用新标记<CreTok>重新定义'创意',让模型理解组合语义。
- 在多个概念上实现直接生成,人类偏好评分更高。
- 基于自适应提示优化,无需额外训练即可通用使用。
‘创意’对人类和扩散模型而言仍是抽象概念。尽管文本到图像扩散模型能轻松生成如‘蓝色香蕉’这类分布外概念,但在生成‘像生菜与螳螂的创意混合体’等组合对象时仍存在困难,根源在于对‘创意’语义深度理解不足。现有方法依赖合成参考提示或图像来实现创意效果,通常需为每种输出重新训练,计算成本高且难以实用。为此,我们提出 CreTok,通过将‘创意’重新定义为新标记 <CreTok>,赋予扩散模型元创意能力。CreTok 利用自建的 CangJie 数据集迭代采样多样文本对,构建自适应提示与约束提示,并优化其文本嵌入间的相似性。大量实验表明,<CreTok> 能在不进行额外训练的前提下,通用且直接生成跨概念的组合创意,在文本-图像对齐与人类偏好评分上均达到当前最优表现。代码将开源于 https://github.com/fu-feng/CreTok。
原文摘要 · Abstract (English)
``Creative'' remains an inherently abstract concept for both humans and diffusion models. While text-to-image (T2I) diffusion models can easily generate out-of-distribution concepts like ``a blue banana'', they struggle with generating combinatorial objects such as ``a creative mixture that resembles a lettuce and a mantis'', due to difficulties in understanding the semantic depth of ``creative''. Current methods rely heavily on synthesizing reference prompts or images to achieve a creative effect, typically requiring retraining for each unique creative output-a process that is computationally intensive and limits practical applications. To address this, we introduce CreTok, which brings meta-creativity to diffusion models by redefining ``creative'' as a new token, \texttt{<CreTok>}, thus enhancing models' semantic understanding for combinatorial creativity. CreTok achieves such redefinition by iteratively sampling diverse text pairs from our proposed CangJie dataset to form adaptive prompts and restrictive prompts, and then optimizing the similarity between their respective text embeddings. Extensive experiments demonstrate that <CreTok> enables the universal and direct generation of combinatorial creativity across diverse concepts without additional training, achieving state-of-the-art performance with improved text-image alignment and higher human preference ratings. Code will be made available at https://github.com/fu-feng/CreTok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。