通过分层缩放补丁大小，提升视觉生成模型的推理速度。
Pyramidal Patchification Flow for Visual Generation
- 在不同噪声阶段使用大小不同的补丁，动态调整计算量。
- 相比标准方法，推理速度提升1.6至2.0倍，训练开销略低。
- 无需重采样技巧，适配预训练模型快速迁移使用。
扩散变换器（DiTs）采用补丁化（Patchify）将补丁表示映射为标记表示，通过线性投影调整输入到DiT模块的标记数量，从而控制计算成本。本文提出分层补丁化流（PPFlow）：在高噪声时间步使用大补丁，低噪声时间步使用小补丁；为每种补丁尺寸学习独立的线性投影；并相应修改解补丁化（Unpatchify）过程。与分层流不同，本方法作用于完整的潜在表示而非金字塔表示，并采用标准去噪过程，无需重采样技巧。通过两种训练方式验证有效性：从头训练使2级和3级分层补丁化在保持相近生成性能下分别实现1.6倍和2.0倍的推理加速，且训练浮点运算量略低；从预训练标准DiTs继续训练则效果更优，训练时间短。代码与检查点见https://github.com/fudan-generative-vision/PPFlow。
原文摘要 · Abstract (English)
Diffusion transformers (DiTs) adopt Patchify, mapping patch representations to token representations through linear projections, to adjust the number of tokens input to DiT blocks and thus the computation cost. Instead of a single patch size for all the timesteps, we introduce a Pyramidal Patchification Flow (PPFlow) approach: Large patch sizes are used for high noise timesteps and small patch sizes for low noise timesteps; Linear projections are learned for each patch size; and Unpatchify is accordingly modified. Unlike Pyramidal Flow, our approach operates over full latent representations other than pyramid representations, and adopts the normal denoising process without requiring the renoising trick. We demonstrate the effectiveness of our approach through two training manners. Training from scratch achieves a $1.6\times$ ($2.0\times$) inference speed over SiT-B/2 for 2-level (3-level) pyramid patchification with slightly lower training FLOPs and similar image generation performance. Training from pretrained normal DiTs achieves even better performance with small training time. The code and checkpoint are at https://github.com/fudan-generative-vision/PPFlow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。