无需训练即可提升扩散Transformer的高清生成质量
ResDiT: Evoking the Intrinsic Resolution Scalability in Diffusion Transformers
- 通过修正位置编码实现分辨率自适应，避免布局错乱
- 融合全局与局部注意力，显著改善细节清晰度
- 适用于可控图像生成等下游任务，兼容现有模型
利用预训练扩散Transformer（DiTs）进行高分辨率（HR）图像生成时，常出现空间布局坍缩和纹理保真度下降问题。以往方法需复杂流程，先在基础分辨率下去噪再引导高清生成。本文提出ResDiT，一种无需训练的高效分辨率扩展方法。研究发现，原始位置编码（PEs）在高分辨率外推时包含错误位置信息，导致布局坍缩。为此，提出PE缩放技术以纠正分辨率变化下的编码偏差；同时设计基于基础分辨率局部注意力的增强机制，通过补丁级融合模块聚合全局与局部线索，并采用高斯加权拼接策略消除网格伪影。大量实验表明，ResDiT持续生成高质量、高分辨率图像，且可无缝集成至下游任务，如空间可控生成。
原文摘要 · Abstract (English)
Leveraging pre-trained Diffusion Transformers (DiTs) for high-resolution (HR) image synthesis often leads to spatial layout collapse and degraded texture fidelity. Prior work mitigates these issues with complex pipelines that first perform a base-resolution (i.e., training-resolution) denoising process to guide HR generation. We instead explore the intrinsic generative mechanisms of DiTs and propose ResDiT, a training-free method that scales resolution efficiently. We identify the core factor governing spatial layout, position embeddings (PEs), and show that the original PEs encode incorrect positional information when extrapolated to HR, which triggers layout collapse. To address this, we introduce a PE scaling technique that rectifies positional encoding under resolution changes. To further remedy low-fidelity details, we develop a local-enhancement mechanism grounded in base-resolution local attention. We design a patch-level fusion module that aggregates global and local cues, together with a Gaussian-weighted splicing strategy that eliminates grid artifacts. Comprehensive evaluations demonstrate that ResDiT consistently delivers high-fidelity, high-resolution image synthesis and integrates seamlessly with downstream tasks, including spatially controlled generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。