动态调整图像不同区域的压缩率，提升生成图像的细节真实感与全局一致性。
D$^2$iT: Dynamic Diffusion Transformer for Accurate Image Generation
- 按区域信息密度动态调整压缩比例，分层编码更精准。
- 多粒度噪声预测实现粗糙与精细区域协同优化，质量显著提升。
- 适合追求高保真图像生成的开发者与研究者使用。
扩散模型在生成高质量图像方面表现优异。尽管扩散Transformer（DiT）架构性能卓越且可扩展，但在扩散过程中对不同图像区域采用固定压缩，忽视了区域间天然存在的信息密度差异。过大的压缩导致局部真实感不足，过小的压缩则增加计算开销并损害全局一致性，最终影响生成质量。为此，我们提出一种动态压缩策略，通过识别区域重要性来调整压缩程度，构建两阶段框架以提升生成效率与效果：(1) 第一阶段动态变分自编码器（DVAE）使用分层编码器以不同下采样率对不同区域编码，适配其信息密度，为扩散过程提供更准确自然的潜在表示；(2) 第二阶段动态扩散Transformer（D²iT）通过动态粒度变换器与动态内容变换器结合，预测多粒度噪声——平滑区域使用粗粒度（较少潜在码），细节区域使用细粒度（较多潜在码），实现全局一致性与局部真实性的统一。多种生成任务上的实验证明了该方法的有效性。代码将发布于 https://github.com/jiawn-creator/Dynamic-DiT。
原文摘要 · Abstract (English)
Diffusion models are widely recognized for their ability to generate high-fidelity images. Despite the excellent performance and scalability of the Diffusion Transformer (DiT) architecture, it applies fixed compression across different image regions during the diffusion process, disregarding the naturally varying information densities present in these regions. However, large compression leads to limited local realism, while small compression increases computational complexity and compromises global consistency, ultimately impacting the quality of generated images. To address these limitations, we propose dynamically compressing different image regions by recognizing the importance of different regions, and introduce a novel two-stage framework designed to enhance the effectiveness and efficiency of image generation: (1) Dynamic VAE (DVAE) at first stage employs a hierarchical encoder to encode different image regions at different downsampling rates, tailored to their specific information densities, thereby providing more accurate and natural latent codes for the diffusion process. (2) Dynamic Diffusion Transformer (D$^2$iT) at second stage generates images by predicting multi-grained noise, consisting of coarse-grained (less latent code in smooth regions) and fine-grained (more latent codes in detailed regions), through an novel combination of the Dynamic Grain Transformer and the Dynamic Content Transformer. The strategy of combining rough prediction of noise with detailed regions correction achieves a unification of global consistency and local realism. Comprehensive experiments on various generation tasks validate the effectiveness of our approach. Code will be released at https://github.com/jiawn-creator/Dynamic-DiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。