arXiv:2505.11497cs.CV2025-05中稿 · ICLR被引 20

4-bit视频生成模型突破,实现与全精度相当的画质。

QVGen: Pushing the Limit of Quantized Video Generative Models

  • 设计新训练框架,用辅助模块缓解低比特量化误差
  • 3比特下在VBench上动态度提升25.28,场景一致性+8.43
  • 训练后自动移除额外模块,推理零开销,适合部署

视频扩散模型(DMs)已实现高质量视频生成,但其巨大的计算与内存需求限制了实际应用,即使在高端GPU上也面临挑战。虽然量化在图像生成模型中取得显著成效,但直接应用于视频模型效果不佳。本文提出QVGen,一种针对极低比特(如4比特或以下)量化设计的量化感知训练(QAT)框架,旨在实现高性能且推理高效的视频生成模型。我们通过理论分析发现,降低梯度范数对QAT收敛至关重要。为此,引入辅助模块Φ以缓解大量化误差,显著提升收敛性。为消除Φ在推理时的开销,提出基于奇异值分解(SVD)和秩正则化γ的降秩策略,逐步剔除贡献小的组件。在4个参数量从1.3B到14B的SOTA视频生成模型上验证,QVGen是首个在4比特设置下达到全精度相当质量的方法,并显著优于现有方案。例如,3比特的CogVideoX-2B在VBench上动态度提升25.28,场景一致性提升8.43。代码与模型已开源。

原文摘要 · Abstract (English)

Video diffusion models (DMs) have enabled high-quality video synthesis. Yet, their substantial computational and memory demands pose serious challenges to real-world deployment, even on high-end GPUs. As a commonly adopted solution, quantization has proven notable success in reducing cost for image DMs, while its direct application to video DMs remains ineffective. In this paper, we present QVGen, a novel quantization-aware training (QAT) framework tailored for high-performance and inference-efficient video DMs under extremely low-bit quantization (e.g., 4-bit or below). We begin with a theoretical analysis demonstrating that reducing the gradient norm is essential to facilitate convergence for QAT. To this end, we introduce auxiliary modules ($Φ$) to mitigate large quantization errors, leading to significantly enhanced convergence. To eliminate the inference overhead of $Φ$, we propose a rank-decay strategy that progressively eliminates $Φ$. Specifically, we repeatedly employ singular value decomposition (SVD) and a proposed rank-based regularization $\mathbfγ$ to identify and decay low-contributing components. This strategy retains performance while zeroing out additional inference overhead. Extensive experiments across $4$ state-of-the-art (SOTA) video DMs, with parameter sizes ranging from $1.3\text{B}\sim14\text{B}$, show that QVGen is the first to reach full-precision comparable quality under 4-bit settings. Moreover, it significantly outperforms existing methods. For instance, our 3-bit CogVideoX-2B achieves improvements of $+25.28$ in Dynamic Degree and $+8.43$ in Scene Consistency on VBench. Code and models are available at https://github.com/ModelTC/QVGen.

视频生成量化扩散模型低比特

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。