让文本嵌入随生成过程动态调整，提升图像与文本的对齐效果。
Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
- 在每步去噪时动态优化文本嵌入，使其更贴合当前生成图像
- 相比固定文本嵌入，在多概念生成和文本编辑任务中对齐度显著提升
- 无需额外训练，即可增强扩散模型的文本条件适应能力
文本到图像扩散模型依赖预训练文本编码器生成的固定文本嵌入，但这些嵌入在所有扩散步骤中保持不变，限制了其对生成过程的适应性。本文提出扩散自适应文本嵌入（DATE），在每个扩散步骤中基于中间扰动数据动态更新文本嵌入。通过构建优化问题并推导更新规则，使文本嵌入在采样过程中持续优化，以增强文本与预测图像均值之间的对齐和偏好。该方法无需额外训练，即可在扩散采样过程中动态适配文本条件。理论分析与实证结果表明，DATE在保持模型生成能力的同时，在多概念生成与文本引导图像编辑等任务上显著优于固定文本嵌入。代码已开源：https://github.com/aailab-kaist/DATE。
原文摘要 · Abstract (English)
Text-to-image diffusion models rely on text embeddings from a pre-trained text encoder, but these embeddings remain fixed across all diffusion timesteps, limiting their adaptability to the generative process. We propose Diffusion Adaptive Text Embedding (DATE), which dynamically updates text embeddings at each diffusion timestep based on intermediate perturbed data. We formulate an optimization problem and derive an update rule that refines the text embeddings at each sampling step to improve alignment and preference between the mean predicted image and the text. This allows DATE to dynamically adapts the text conditions to the reverse-diffused images throughout diffusion sampling without requiring additional model training. Through theoretical analysis and empirical results, we show that DATE maintains the generative capability of the model while providing superior text-image alignment over fixed text embeddings across various tasks, including multi-concept generation and text-guided image editing. Our code is available at https://github.com/aailab-kaist/DATE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。