通过遮蔽关键查询令牌提升细节生成能力
FocusDiT: Masking Queries in Diffusion Transformers for Fine-grained Image Generation
- 只将重要查询令牌输入前馈网络，聚焦细节生成
- 在文本到图像任务中显著提升生成质量
- 适合需要高精度细节生成的研究与应用
扩散变换器（DiT）在生成扩散模型中广泛应用，通过注意力机制和前馈网络（FFN）实现查询标记的去噪。事实上，FFN充当解码视觉内容的键值词汇表，其中值向量嵌入视觉语义知识。本文提出，关注对应复杂细节的关键查询标记，并鼓励模型优化这些标记，对细粒度视觉生成至关重要。为此，我们提出FocusDiT，采用遮蔽策略，仅将关键查询标记输入FFN。被遮蔽的查询可从FFN词汇表中检索视觉标记，用于解码其细节。大量文本到图像实验验证了该遮蔽策略在提升生成性能方面的有效性。
原文摘要 · Abstract (English)
Diffusion transformer (DiT) has been widely adopted in the generative diffusion field, advancing the denoising of query tokens through attention and Feed-Forward (\text{FFN}) layers. FFN actually acts as the key-value vocabulary for decoding visual contents where the value embeds the visual semantical knowledge. We present that focusing on critical query tokens corresponding to more complex details and encouraging the model to improve these tokens is essential for fine-grained visual generation. To this end, we propose FocusDiT, which applies a Masking scheme to focus on critical query tokens that are exclusively fed into FFN. The masked queries can retrieve visual tokens from the FFN vocabularies, and use them to decode their visual details. Extensive text-to-image experiments validate the effectiveness of token masking in enhancing generative performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。