无需训练，仅在生成时就能分离控制图像的色彩与风格。
Training-free Color-Style Disentanglement for Constrained Text-to-Image Synthesis
- 生成时通过特征变换使潜在表示的协方差匹配参考图。
- 利用LAB空间中色彩与风格的自然分离特性进行独立控制。
- 支持同一或不同参考图融合色彩与风格，灵活且无需重训练。
我们研究如何在文本到图像扩散模型中，以解耦方式独立控制输出的色彩与风格属性，这些属性来自用户提供的参考图像。本文提出首个无需训练、仅在测试阶段即可实现色彩与风格解耦并条件化的方法。核心创新包括：第一，在推理时对潜在编码进行特征变换，使当前生成结果的协方差矩阵与参考图像一致，有效传递色彩信息；第二，观察到在LAB图像空间中色彩与风格存在天然解耦，因此利用参考图像的L通道计算自注意力特征映射，并将其应用于生成图像的对应特征，实现风格迁移。上述操作均在测试阶段完成，可独立或合并使用。该方法支持从同一参考图或两个不同来源分别获取色彩与风格信息，新生成图像可无缝融合二者，具备高度灵活性。
原文摘要 · Abstract (English)
We consider the problem of independently, in a disentangled fashion, controlling the outputs of text-to-image diffusion models with color and style attributes of a user-supplied reference image. We present the first training-free, test-time-only method to disentangle and condition text-to-image models on color and style attributes from reference image. To realize this, we propose two key innovations. Our first contribution is to transform the latent codes at inference time using feature transformations that make the covariance matrix of current generation follow that of the reference image, helping meaningfully transfer color. Next, we observe that there exists a natural disentanglement between color and style in the LAB image space, which we exploit to transform the self-attention feature maps of the image being generated with respect to those of the reference computed from its L channel. Both these operations happen purely at test time and can be done independently or merged. This results in a flexible method where color and style information can come from the same reference image or two different sources, and a new generation can seamlessly fuse them in either scenario.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。