通过剪枝大幅降低音频生成模型计算量,同时保持甚至提升音质。
Efficient Text-to-Audio Generation via Pruning

- 对扩散模型的U-Net结构进行基于范数的滤波器剪枝
- 参数减少83%,计算量降低39%,音质不降反升
- 轻量微调可恢复剪枝导致的关键声音生成能力损失
基于扩散的文生音频模型如AudioLDM虽能生成高质量、语义一致的音频,但其计算成本高,制约实际部署。本文针对AudioLDM这一基于U-Net的文本条件音频潜在扩散模型,采用模型剪枝技术提升效率。通过分析U-Net各卷积模块的参数冗余性,提出基于范数的滤波器剪枝策略,并结合轻量微调恢复性能损失。实验表明,剪枝后可实现高达83%的参数减少和39%的乘加操作降低,且在多数情况下生成质量优于原始未剪枝模型。然而,剪枝会影响枪声、警笛、爆炸等安全关键音效以及电钻、缝纫机等机械声、喷雾声、滴答声等生成能力,这些缺陷可通过轻量微调有效恢复。
原文摘要 · Abstract (English)
Diffusion-based text-to-audio generative models such as AudioLDM achieve high perceptual quality and strong semantic consistency; however, their practical deployment is hindered by the substantial computational cost of the U-Net denoising backbone. In this work, we apply model pruning to improve the computational efficiency of AudioLDM, a U-Net-based text-conditioned audio latent diffusion model. We analyse parameter redundancy across U-Net convolutional blocks and evaluate a filter-pruning strategy. Pruning is guided by norm-based criteria and followed by lightweight finetuning to recover performance losses. Experimental results demonstrate that up to 83% of the parameters and 39% of the multiply-accumulate operations of U-Net have been reduced while maintaining, and in some cases improving, generation quality compared to the baseline unpruned network. We find that pruning affects AudioLDM's ability to generate certain sound events including safety-critical sounds such as gunshots, sirens, and explosions, as well as mechanical sounds such as drills and sewing machines, and other sounds such as sprays and tick-tocks, which are mostly recovered by lightweight finetuning of the pruned model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。