将完整文本拆解分阶段注入,提升文生图模型对语义细节的理解能力。
Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning
- 把完整描述拆成简化短句,分层逐步注入模型
- 在不同去噪阶段精准注入特定语义信息,提升细节表达
- 适合需要精细控制生成内容的图像创作场景
当前文生图扩散模型多采用完整文本条件输入,但因语法复杂,扩散变压器(DiT)存在理解缺陷。一次性输入完整文本可能忽略关键语义或造成语义混淆。为此,本文提出新框架DiT-ST,将完整文本转换为一组简化短句组成的拆分文本,显式表达各类语义原语及其关联。拆分文本以分层递进方式注入DiT-ST的不同去噪阶段。具体地,利用大语言模型解析文本,提取并分层组织多种语义原语构建拆分输入;同时根据不同语义原语对去噪过程的敏感度差异,划分扩散去噪流程,并确定合适时间步通过交叉注意力逐项注入对应语义原语的标记。该方法显著增强模型在各阶段对特定语义原语的表征学习能力。大量实验证明,DiT-ST有效缓解了完整文本理解缺陷。
原文摘要 · Abstract (English)
Current text-to-image diffusion generation typically employs complete-text conditioning. Due to the intricate syntax, diffusion transformers (DiTs) inherently suffer from a comprehension defect of complete-text captions. One-fly complete-text input either overlooks critical semantic details or causes semantic confusion by simultaneously modeling diverse semantic primitive types. To mitigate this defect of DiTs, we propose a novel split-text conditioning framework named DiT-ST. This framework converts a complete-text caption into a split-text caption, a collection of simplified sentences, to explicitly express various semantic primitives and their interconnections. The split-text caption is then injected into different denoising stages of DiT-ST in a hierarchical and incremental manner. Specifically, DiT-ST leverages Large Language Models to parse captions, extracting diverse primitives and hierarchically sorting out and constructing these primitives into a split-text input. Moreover, we partition the diffusion denoising process according to its differential sensitivities to diverse semantic primitive types and determine the appropriate timesteps to incrementally inject tokens of diverse semantic primitive types into input tokens via cross-attention. In this way, DiT-ST enhances the representation learning of specific semantic primitive types across different stages. Extensive experiments validate the effectiveness of our proposed DiT-ST in mitigating the complete-text comprehension defect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。