无需训练,在生成时用颜色/边缘图控制图像输出。
Test-time Conditional Text-to-Image Synthesis Using Diffusion Models
- 生成时直接操控扩散模型的噪声,不修改模型参数。
- 首次实现用颜色分布匹配控制生成图像色彩,性能显著提升。
- 适合需要灵活调整图像风格的研究者和开发者。
我们研究了基于扩散模型的条件文本到图像生成问题。现有方法通常需要微调模型特定部分或引入可训练参数,导致部署灵活性差。为此,我们提出TINTIN:一种仅在测试时运行、无需训练的新方法,可将颜色配色方案和边缘图等条件因素用于控制扩散模型的输出。具体而言,我们将去噪过程中的噪声预测视为能量模型的梯度,通过匹配由这些预测推导出的输出与真实条件输入,实现对噪声的灵活操控。这使得本方法在已知条件下,首次实现以颜色分布匹配损失控制生成图像的色彩。此外,该测试时噪声操控方法可轻松扩展至其他条件类型(如边缘图)。我们在多种文本提示、颜色配色方案和边缘图上进行了广泛实验,结果表明该方法在定性和定量指标上均显著优于当前最先进水平。
原文摘要 · Abstract (English)
We consider the problem of conditional text-to-image synthesis with diffusion models. Most recent works need to either finetune specific parts of the base diffusion model or introduce new trainable parameters, leading to deployment inflexibility due to the need for training. To address this gap in the current literature, we propose our method called TINTIN: Test-time Conditional Text-to-Image Synthesis using Diffusion Models which is a new training-free test-time only algorithm to condition text-to-image diffusion model outputs on conditioning factors such as color palettes and edge maps. In particular, we propose to interpret noise predictions during denoising as gradients of an energy-based model, leading to a flexible approach to manipulate the noise by matching predictions inferred from them to the ground truth conditioning input. This results in, to the best of our knowledge, the first approach to control model outputs with input color palettes, which we realize using a novel color distribution matching loss. We also show this test-time noise manipulation can be easily extensible to other types of conditioning, e.g., edge maps. We conduct extensive experiments using a variety of text prompts, color palettes, and edge maps and demonstrate significant improvement over the current state-of-the-art, both qualitatively and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。