提出PTQ4ADM框架,让音频扩散模型压缩70%体积且保持高质量生成。
PTQ4ADM: Post-Training Quantization for Efficient Text Conditional Audio Diffusion Models
- 通过提示增强和激活感知校准,提升量化覆盖度与保真度。
- 关键层可低至4比特权重+8比特激活,质量损失小于5%。
- 适合部署在资源受限设备上的文本到音频生成应用。
去噪扩散模型在图像、音频和视频生成任务中表现卓越,但其高计算成本和大内存占用限制了广泛应用。后训练量化(PTQ)可通过降低参数带宽缓解此问题,然而直接应用于扩散模型会导致多步去噪过程中量化噪声累积,尤其在文本条件生成任务中影响显著。本文提出PTQ4ADM,一种针对音频扩散模型(ADMs)的新量化框架,核心贡献包括:(1) 覆盖驱动的提示增强方法,(2) 针对文本条件ADM的激活感知校准集生成算法。这些技术确保全面覆盖音频特性与模态,同时保持合成质量。我们在TANGO、Make-An-Audio和AudioLDM模型上验证该方法,实验表明可将模型大小减少高达70%,合成质量指标与全精度模型相当(FD分数增加小于5%)。我们还发现,骨干网络中的特定层可被量化至4比特权重和8比特激活而无明显质量下降。该工作为资源受限环境下的音频扩散模型高效部署铺平道路。
原文摘要 · Abstract (English)
Denoising diffusion models have emerged as state-of-the-art in generative tasks across image, audio, and video domains, producing high-quality, diverse, and contextually relevant data. However, their broader adoption is limited by high computational costs and large memory footprints. Post-training quantization (PTQ) offers a promising approach to mitigate these challenges by reducing model complexity through low-bandwidth parameters. Yet, direct application of PTQ to diffusion models can degrade synthesis quality due to accumulated quantization noise across multiple denoising steps, particularly in conditional tasks like text-to-audio synthesis. This work introduces PTQ4ADM, a novel framework for quantizing audio diffusion models(ADMs). Our key contributions include (1) a coverage-driven prompt augmentation method and (2) an activation-aware calibration set generation algorithm for text-conditional ADMs. These techniques ensure comprehensive coverage of audio aspects and modalities while preserving synthesis fidelity. We validate our approach on TANGO, Make-An-Audio, and AudioLDM models for text-conditional audio generation. Extensive experiments demonstrate PTQ4ADM's capability to reduce the model size by up to 70\% while achieving synthesis quality metrics comparable to full-precision models($<$5\% increase in FD scores). We show that specific layers in the backbone network can be quantized to 4-bit weights and 8-bit activations without significant quality loss. This work paves the way for more efficient deployment of ADMs in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。