不改架构让扩散Transformer训练快7倍，精度接近原模型。
FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
- 通过分析噪声比分布，揭示训练策略失效本质。
- 提出新监督方法，100万步达2.30 FID，速度提升7倍。
- 仅需少量代码修改，适合追求训练效率的研究者。
扩散Transformer（DiT）在研究中备受关注，但其训练收敛缓慢。本文旨在不修改网络结构的前提下加速DiT训练。我们发现现有训练策略在不同数据上表现不稳定，且特定时间步的监督效果有限。为此，我们提出三点贡献：(1) 略微扩展信噪比（SNR）定义，通过观察SNR的概率密度函数（PDF）来理解策略的数据鲁棒性；(2) 基于大量实验（超100组），从PDF视角归纳出统一的加速策略；(3) 设计一种新型监督机制，进一步加速训练。基于此，我们提出FasterDiT，一种简单实用的设计方案。仅需少量代码修改，即可在100万次迭代下于ImageNet 256分辨率上达到2.30 FID，与原版DiT（2.27 FID）相当，但训练速度提升7倍。
原文摘要 · Abstract (English)
Diffusion Transformers (DiT) have attracted significant attention in research. However, they suffer from a slow convergence rate. In this paper, we aim to accelerate DiT training without any architectural modification. We identify the following issues in the training process: firstly, certain training strategies do not consistently perform well across different data. Secondly, the effectiveness of supervision at specific timesteps is limited. In response, we propose the following contributions: (1) We introduce a new perspective for interpreting the failure of the strategies. Specifically, we slightly extend the definition of Signal-to-Noise Ratio (SNR) and suggest observing the Probability Density Function (PDF) of SNR to understand the essence of the data robustness of the strategy. (2) We conduct numerous experiments and report over one hundred experimental results to empirically summarize a unified accelerating strategy from the perspective of PDF. (3) We develop a new supervision method that further accelerates the training process of DiT. Based on them, we propose FasterDiT, an exceedingly simple and practicable design strategy. With few lines of code modifications, it achieves 2.30 FID on ImageNet 256 resolution at 1000k iterations, which is comparable to DiT (2.27 FID) but 7 times faster in training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。