动态调整图像不同区域采样频率,实现扩散模型加速
Region-Adaptive Sampling for Diffusion Transformers
- 按模型关注区域动态分配采样比例,非均匀更新
- 在稳定扩散3和Lumina-Next-T2I上分别提速2.36倍和2.51倍
- 无需训练,适合追求实时生成的视觉应用开发者
扩散模型已成为各类生成任务的主流选择,但其依赖多步顺序前向传播严重限制了实时性能。以往加速方法主要聚焦减少采样步数或复用中间结果,未能利用图像各空间区域的差异,这受限于卷积U-Net结构。通过利用扩散Transformer(DiT)对可变数量标记的灵活性,我们提出RAS——一种无需训练的采样策略,根据DiT模型的关注区域动态分配不同采样比例。关键观察发现:每一步采样中,模型聚焦于语义有意义区域,且这些关注区域在连续步骤间具有强时间一致性。基于此,RAS仅更新当前关注区域,其余区域使用上一步缓存的噪声。模型关注区域由前一步输出决定,充分利用了观测到的时间一致性。我们在Stable Diffusion 3和Lumina-Next-T2I上评估RAS,分别实现最高2.36倍和2.51倍加速,生成质量下降极小。用户研究显示,人类评价下RAS生成质量相当,同时获得1.6倍速度提升。该方法显著推进了高效扩散Transformer的发展,增强了其实时应用潜力。
原文摘要 · Abstract (English)
Diffusion models (DMs) have become the leading choice for generative tasks across diverse domains. However, their reliance on multiple sequential forward passes significantly limits real-time performance. Previous acceleration methods have primarily focused on reducing the number of sampling steps or reusing intermediate results, failing to leverage variations across spatial regions within the image due to the constraints of convolutional U-Net structures. By harnessing the flexibility of Diffusion Transformers (DiTs) in handling variable number of tokens, we introduce RAS, a novel, training-free sampling strategy that dynamically assigns different sampling ratios to regions within an image based on the focus of the DiT model. Our key observation is that during each sampling step, the model concentrates on semantically meaningful regions, and these areas of focus exhibit strong continuity across consecutive steps. Leveraging this insight, RAS updates only the regions currently in focus, while other regions are updated using cached noise from the previous step. The model's focus is determined based on the output from the preceding step, capitalizing on the temporal consistency we observed. We evaluate RAS on Stable Diffusion 3 and Lumina-Next-T2I, achieving speedups up to 2.36x and 2.51x, respectively, with minimal degradation in generation quality. Additionally, a user study reveals that RAS delivers comparable qualities under human evaluation while achieving a 1.6x speedup. Our approach makes a significant step towards more efficient diffusion transformers, enhancing their potential for real-time applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。