通过量化与注意力稀疏化联合压缩视频生成模型,显著提升效率且保持高质量。
QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification
- 融合量化与注意力稀疏化,用结构引导和局部监督减少误差。
- 在HunyuanVideo-13B上实现20.88 PSNR,优于现有基线16.85 PSNR。
- 适合需要高效部署高分辨率视频生成模型的研究与工程团队。
扩散变换器在视频生成方面表现出色,但其高昂的计算与内存成本限制了实际应用。模型量化和注意力稀疏化是两种有前景的压缩方向,但单独使用在激进压缩下会导致严重性能下降。两者结合虽能带来协同增效,但简单集成效果不佳,稀疏化引起的失真会加剧量化噪声,导致注意力分布显著偏移。为此,我们提出QuantSparse,一个统一框架,整合模型量化与注意力稀疏化。具体地,引入多尺度显著注意力蒸馏,利用全局结构引导与局部显著性监督缓解量化带来的偏差;同时设计二阶稀疏注意力重参数化,利用二阶残差的时间稳定性高效恢复稀疏化损失的信息。在HunyuanVideo-13B上的实验表明,QuantSparse达到20.88 PSNR,显著优于当前最佳量化基线Q-VDiT(16.85 PSNR),同时实现存储量降低3.68倍、端到端推理加速1.88倍。代码将发布于https://github.com/wlfeng0509/QuantSparse。
原文摘要 · Abstract (English)
Diffusion transformers exhibit remarkable video generation capability, yet their prohibitive computational and memory costs hinder practical deployment. Model quantization and attention sparsification are two promising directions for compression, but each alone suffers severe performance degradation under aggressive compression. Combining them promises compounded efficiency gains, but naive integration is ineffective. The sparsity-induced information loss exacerbates quantization noise, leading to amplified attention shifts. To address this, we propose \textbf{QuantSparse}, a unified framework that integrates model quantization with attention sparsification. Specifically, we introduce \textit{Multi-Scale Salient Attention Distillation}, which leverages both global structural guidance and local salient supervision to mitigate quantization-induced bias. In addition, we develop \textit{Second-Order Sparse Attention Reparameterization}, which exploits the temporal stability of second-order residuals to efficiently recover information lost under sparsity. Experiments on HunyuanVideo-13B demonstrate that QuantSparse achieves 20.88 PSNR, substantially outperforming the state-of-the-art quantization baseline Q-VDiT (16.85 PSNR), while simultaneously delivering a \textbf{3.68$\times$} reduction in storage and \textbf{1.88$\times$} acceleration in end-to-end inference. Our code will be released in https://github.com/wlfeng0509/QuantSparse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。