让扩散Transformer动态调整计算量，少算40%还能保持画质
FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute
- 用动态计算策略替代固定算力，按需分配资源
- 生成图像时减少超40%计算量，画质无下降
- 适用于图文和视频生成，适合部署在低算力设备
尽管现代扩散Transformer性能出色，但推理时消耗大量资源，源于每步去噪需固定且庞大的计算量。本文重新审视传统的静态计算分配范式，提出一种动态策略。所提方法简单高效，可将预训练的DiT模型转换为灵活可变的FlexiDiT模型，支持在不同计算预算下处理输入。实验表明，单个灵活模型在类别条件与文本条件图像生成中，相比静态模型降低超过40%的FLOPs，且画质无损。该方法通用且不依赖输入或条件模态，可轻松扩展至视频生成，使生成样本计算量减少高达75%，性能不受影响。
原文摘要 · Abstract (English)
Despite their remarkable performance, modern Diffusion Transformers are hindered by substantial resource requirements during inference, stemming from the fixed and large amount of compute needed for each denoising step. In this work, we revisit the conventional static paradigm that allocates a fixed compute budget per denoising iteration and propose a dynamic strategy instead. Our simple and sample-efficient framework enables pre-trained DiT models to be converted into \emph{flexible} ones -- dubbed FlexiDiT -- allowing them to process inputs at varying compute budgets. We demonstrate how a single \emph{flexible} model can generate images without any drop in quality, while reducing the required FLOPs by more than $40$\% compared to their static counterparts, for both class-conditioned and text-conditioned image generation. Our method is general and agnostic to input and conditioning modalities. We show how our approach can be readily extended for video generation, where FlexiDiT models generate samples with up to $75$\% less compute without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。