动态分配精度与时间缓存,让视频生成模型更快更省内存。
6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models
- 根据层间差异动态分配NVFP4/INT8精度,提升压缩效率。
- 实现1.92倍加速与3.32倍内存减少,质量无损失。
- 适合部署视频扩散模型的高效推理场景。
扩散变换器在视频生成方面表现出色,但其实际部署受限于高内存占用和计算开销。后训练量化为降低内存使用和提升计算速度提供了可行方案。现有方法通常采用静态位宽分配,忽视了扩散过程中各时间步激活值的量化难易差异,导致效率与质量的权衡不佳。本文提出一种推理时的NVFP4/INT8混合精度量化框架。我们发现模块输入输出差异与其内部线性层的量化敏感性存在强线性相关性,据此设计轻量级预测器,动态将NVFP4分配给时序稳定的层以最大化内存压缩,同时对易变层保留INT8以保证鲁棒性。该自适应策略实现了激进量化而不影响生成质量。此外,观察到变压器块的输入输出残差在时间上高度一致,利用此时间冗余,引入时间差分缓存(TDC)跳过这些不变层的计算,进一步降低计算开销。大量实验表明,本方法实现1.92×端到端加速和3.32×内存减少,为视频扩散变换器的高效推理树立了新基准。
原文摘要 · Abstract (English)
Diffusion transformers have demonstrated remarkable capabilities in generating videos. However, their practical deployment is severely constrained by high memory usage and computational cost. Post-Training Quantization provides a practical way to reduce memory usage and boost computation speed. Existing quantization methods typically apply a static bit-width allocation, overlooking the quantization difficulty of activations across diffusion timesteps, leading to a suboptimal trade-off between efficiency and quality. In this paper, we propose a inference time NVFP4/INT8 Mixed-Precision Quantization framework. We find a strong linear correlation between a block's input-output difference and the quantization sensitivity of its internal linear layers. Based on this insight, we design a lightweight predictor that dynamically allocates NVFP4 to temporally stable layers to maximize memory compression, while selectively preserving INT8 for volatile layers to ensure robustness. This adaptive precision strategy enables aggressive quantization without compromising generation quality. Beside this, we observe that the residual between the input and output of a Transformer block exhibits high temporal consistency across timesteps. Leveraging this temporal redundancy, we introduce Temporal Delta Cache (TDC) to skip computations for these invariant blocks, further reducing the computational cost. Extensive experiments demonstrate that our method achieves 1.92$\times$ end-to-end acceleration and 3.32$\times$ memory reduction, setting a new baseline for efficient inference in Video DiTs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。