用引导式嵌入提升扩散模型文本控制力，无需训练
Rethinking Global Text Conditioning in Diffusion Transformers
- 让文本嵌入作引导而非直接调制，实现无训练控制
- 在图文生成与图像编辑中均显著提升质量
- 方法简单高效，可通用到多种扩散模型
扩散变换器通常通过注意力层和池化文本嵌入的调制机制引入文本信息。然而，近期方法已摒弃调制机制，仅依赖注意力。本文探究调制是否必要及其性能优势。分析表明，传统使用下池化嵌入对整体性能贡献甚微，说明注意力本身已足够传递提示信息。但研究发现，若将池化嵌入以新视角使用——作为引导并实现更理想属性的可控偏移，可带来显著提升。该方法无需训练、实现简单、运行开销极小，适用于多种扩散模型，在文本到图像/视频生成及图像编辑等任务中均取得改进。
原文摘要 · Abstract (English)
Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance, suggesting that attention alone is generally sufficient for faithfully propagating prompt information. However, we reveal that the pooled embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various diffusion models, bringing improvements across diverse tasks, including text-to-image/video generation and image editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。