arXiv:2412.16822cs.CVcs.AI2024-12CVPR被引 27

动态调整图像生成中各层和步骤的计算量,提升效率且不损失质量。

Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers

  • 按像素重要性动态跳过冗余计算,减少无效处理。
  • 不同层级和去噪阶段自动学习压缩比例,最高压缩率达90%以上。
  • 适合部署在手机等低资源设备上,兼顾速度与图像质量。

扩散变换器(DiTs)虽在图像生成质量上达到顶尖水平,但存在延迟高、内存效率低的问题,难以在资源受限设备上部署。现有方法对图像各区域采用均等计算,然而并非所有图像标记都同等重要,例如物体区域需要更多计算。为此,本文提出DiffCR,一种带可微压缩比的动态推理框架,能自动学习在不同层和时间步对每个图像标记进行计算路由,实现高效DiTs。具体包括:(1) 基于标记级别的路由机制,每层包含一个与模型权重联合优化的路由器,预测标记重要性,不重要的标记直接跳过整层计算;(2) 层级可微压缩比机制,各层从零初始化自动学习不同压缩率,冗余层压缩比可达90%,其他层保持低压缩甚至不压缩;(3) 时间步可微压缩比机制,去噪初期噪声大时压缩比更高,图像逐渐清晰后降低压缩比。在文本到图像生成与图像修复任务上的大量实验表明,DiffCR有效捕捉了标记、层和时间步三轴的动态特性,相比以往工作实现了更优的质量-效率权衡。项目主页见https://www.haoranyou.com/diffcr。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) image generation quality but suffer from high latency and memory inefficiency, making them difficult to deploy on resource-constrained devices. One major efficiency bottleneck is that existing DiTs apply equal computation across all regions of an image. However, not all image tokens are equally important, and certain localized areas require more computation, such as objects. To address this, we propose DiffCR, a dynamic DiT inference framework with differentiable compression ratios, which automatically learns to dynamically route computation across layers and timesteps for each image token, resulting in efficient DiTs. Specifically, DiffCR integrates three features: (1) A token-level routing scheme where each DiT layer includes a router that is fine-tuned jointly with model weights to predict token importance scores. In this way, unimportant tokens bypass the entire layer's computation; (2) A layer-wise differentiable ratio mechanism where different DiT layers automatically learn varying compression ratios from a zero initialization, resulting in large compression ratios in redundant layers while others remain less compressed or even uncompressed; (3) A timestep-wise differentiable ratio mechanism where each denoising timestep learns its own compression ratio. The resulting pattern shows higher ratios for noisier timesteps and lower ratios as the image becomes clearer. Extensive experiments on text-to-image and inpainting tasks show that DiffCR effectively captures dynamism across token, layer, and timestep axes, achieving superior trade-offs between generation quality and efficiency compared to prior works. The project website is available at https://www.haoranyou.com/diffcr.

扩散模型高效推理动态压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。