通过自适应剪枝让扩散模型更轻量,还能提升生成效果。
PeLAP-A: Adaptive Latent Pruning for Lightweight Latent Diffusion Models

- 引入可学习的通道重要性预测器,动态抑制低效特征通道。
- 在0.01稀疏性正则下,生成损失降为0.0236,重建误差更低。
- 揭示了扩散模型对通道剪枝有惊人鲁棒性,适合高效部署场景。
潜在扩散模型通过变分自编码器(VAE)生成的压缩潜在空间实现强大的生成性能。然而,尚不清楚所有潜在通道是否对扩散过程贡献均等,或是否存在显著冗余。我们提出PeLAP-A(自适应潜在剪枝用于扩散),一个轻量级框架,在标准潜在扩散流程中加入可学习的通道级重要性预测器。该预测器由两层MLP构成,基于全局池化后的潜在特征生成软掩码,在进入去噪UNet前抑制不重要通道。整个系统在CIFAR-10上联合训练,采用扩散、重构与稀疏性损失。实验发现:在强稀疏性正则(λ=0.01)下,重要性预测器使所有通道趋近于零,但去噪UNet的扩散损失降至0.0236(基线0.0240),VAE重构均方误差降至22.59(基线24.67)。这一现象被称为稀疏性坍缩,我们分析其成因并揭示潜在扩散模型的信息需求特性。本研究探索了潜在扩散训练中的稀疏性动态,表明去噪网络对通道抑制具有极强鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Latent diffusion models achieve strong generative performance by operating in a compressed latent space produced by a variational autoencoder (VAE). However, it remains unclear whether all latent channels contribute equally to the diffusion process, or whether significant redundancy exists. We introduce PeLAP-A (Adaptive Latent Pruning for Diffusion), a lightweight framework that augments a standard latent diffusion pipeline with a learnable channel-wise importance predictor. A two-layer MLP operating on globally pooled latent features produces a soft mask that suppresses unimportant latent channels before they enter the denoising UNet. The entire system is trained jointly on CIFAR-10 under a combined diffusion, reconstruction, and sparsity loss. Experiments reveal a striking result: under aggressive sparsity regularization (lambda = 0.01), the importance predictor drives all latent channels to near-zero yet the denoising UNet achieves lower diffusion loss (0.0236 vs. 0.0240) and lower VAE reconstruction MSE (22.59 vs. 24.67) compared to the unpruned baseline. We term this the sparsity collapse phenomenon and provide an analysis of why it occurs and what it reveals about the information requirements of latent diffusion models. These findings constitute an exploratory study of sparsity dynamics in latent diffusion training, and demonstrate that denoising UNets can remain remarkably robust to latent channel suppression even under aggressive regularization. Code is available at: https://github.com/kissasium/PeLAP-A.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。