通过优化时空冗余，让扩散Transformer生成图像更快更省算力。
RSTR: Reducing SpatioTemporal Redundancy in Diffusion Transformers
- 用进化搜索找出稀疏的引导调度策略，动态调整关键步骤。
- 根据各模块敏感度差异，自适应分配校准资源，减少无效计算。
- 实测节省50%-70%算力，部分模型速度提升3.4倍且质量不降反升。
扩散Transformer（DiTs）在图像生成中表现优异，但部署受高计算成本制约。我们发现两大冗余来源：一是时间冗余——分类器无关引导（CFG）在每一步执行双前向传播，但引导仅在特定步骤重要，关键步骤采用可变尺度可补偿跳过其他步骤；二是空间冗余——不同注意力块对引导响应不均，统一校准既浪费算力又无法满足差异需求。本文提出RSTR，首个联合降低扩散变压器时空冗余的框架。第一阶段通过进化搜索发现具有可变尺度的稀疏引导调度；第二阶段通过自适应秩分配，按敏感度为不同区域分配校准能力。在DiT-XL/2、PixArt-α、FLUX及前沿Qwen-Image模型上，实现50%-70%算力节省，同时保持或提升生成质量。以DiT-XL/2为例，节省57%算力且FID降低15%；在Qwen-Image上实现3.43×加速，质量不变。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have achieved remarkable success in image generation, yet their deployment is hindered by high computational costs. We identify two sources of redundancy. First, temporal redundancy: Classifier-Free Guidance (CFG) applies costly dual forward passes at every timestep, yet guidance matters only at specific steps, and variable scales at critical steps can compensate for skipping others. Second, spatial redundancy: under variable guidance, different transformer blocks exhibit heterogeneous sensitivity, yet uniform calibration across all blocks wastes computation while failing to address their varying requirements. We present RSTR, the first framework to jointly reduce spatiotemporal redundancy in diffusion transformers. Stage-1 addresses temporal redundancy through evolutionary search, discovering sparse guidance schedules with variable scales. Stage-2 addresses spatial redundancy through adaptive rank allocation, assigning calibration capacities to transformer regions based on their sensitivity. Experiments on DiT-XL/2, PixArt-$α$, FLUX, and state-of-the-art Qwen-Image demonstrate 50%-70% compute savings while maintaining or improving quality. On DiT-XL/2, RSTR achieves 57% savings with 15% FID improvement; on Qwen-Image, 3.43$\times$ speedup with preserved quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。