提出S²Q-VDiT,让视频生成模型在极低精度下仍保持高质量且加速3.9倍。
S$^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation
- 用显著数据与稀疏注意力蒸馏,解决视频扩散模型量化难题
- W4A6量化下零性能损失,模型压缩3.9倍,推理提速1.3倍
- 适合需部署轻量级视频生成模型的研究者与开发者
扩散变压器已成为视频生成的主流范式,但其参数量高达数十亿,计算开销巨大。量化可有效降低内存占用并加速推理,然而我们发现,视频扩散模型(V-DMs)中时空联合建模导致极长的令牌序列,引发高校准方差与学习挑战。为此,我们提出S²Q-VDiT,一种针对V-DMs的后训练量化框架,融合显著数据与稀疏令牌蒸馏。校准阶段,提出基于海森矩阵的显著数据选择,结合扩散与量化特性构建高质量校准集;为应对学习挑战,分析了模型固有的稀疏注意力模式,提出注意力引导的稀疏令牌蒸馏,利用令牌级注意力分布强化对输出影响更大的令牌。在W4A6量化下,该方法实现零性能损失,模型压缩3.9倍,推理速度提升1.3倍。代码将开源于https://github.com/wlfeng0509/s2q-vdit。
原文摘要 · Abstract (English)
Diffusion transformers have emerged as the mainstream paradigm for video generation models. However, the use of up to billions of parameters incurs significant computational costs. Quantization offers a promising solution by reducing memory usage and accelerating inference. Nonetheless, we observe that the joint modeling of spatial and temporal information in video diffusion models (V-DMs) leads to extremely long token sequences, which introduces high calibration variance and learning challenges. To address these issues, we propose S$^2$Q-VDiT, a post-training quantization framework for V-DMs that leverages Salient data and Sparse token distillation. During the calibration phase, we identify that quantization performance is highly sensitive to the choice of calibration data. To mitigate this, we introduce \textit{Hessian-aware Salient Data Selection}, which constructs high-quality calibration datasets by considering both diffusion and quantization characteristics unique to V-DMs. To tackle the learning challenges, we further analyze the sparse attention patterns inherent in V-DMs. Based on this observation, we propose \textit{Attention-guided Sparse Token Distillation}, which exploits token-wise attention distributions to emphasize tokens that are more influential to the model's output. Under W4A6 quantization, S$^2$Q-VDiT achieves lossless performance while delivering $3.9\times$ model compression and $1.3\times$ inference acceleration. Code will be available at https://github.com/wlfeng0509/s2q-vdit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。