发现稳定扩散模型可高效压缩，38.5%稀疏度下仍保高质量。
Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion
- 分别对文本编码器和图像生成器进行剪枝，探索不同稀疏度下的效果。
- 在38.5%稀疏度下模型质量损失极小，推理速度提升显著。
- 剪枝后性能突降现象揭示关键权重承载语义信息，适合模型优化研究者参考。
随着文生图模型日益强大且复杂，其庞大的规模成为资源受限设备广泛部署的障碍。本文首次系统研究了Stable Diffusion 2的训练后剪枝，聚焦多模态生成模型的压缩需求。我们分别评估剪枝整个模型或单一组件在不同稀疏度下的表现。结果表明：与语言模型剪枝趋势相反，简单基于幅度的剪枝在文生图任务中表现更优。模型可被剪至38.5%稀疏度而保持高质量输出。最优配置为文本编码器剪至47.5%、扩散生成器剪至35%。该配置大幅降低计算开销并维持图像生成质量。此外，我们观察到剪枝超过阈值后出现性能骤降（图像无法识别），暗示特定权重编码关键语义信息。这一发现为模型压缩、跨模态互操作性及偏见识别提供了新方向。
原文摘要 · Abstract (English)
As text-to-image models grow increasingly powerful and complex, their burgeoning size presents a significant obstacle to widespread adoption, especially on resource-constrained devices. This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain. Our study tackles the pruning techniques for the previously unexplored multi-modal generation models, and particularly examines the pruning impact on the textual component and the image generation component separately. We conduct a comprehensive comparison on pruning the model or the single component of the model in various sparsities. Our results yield previously undocumented findings. For example, contrary to established trends in language model pruning, we discover that simple magnitude pruning outperforms more advanced techniques in text-to-image context. Furthermore, our results show that Stable Diffusion 2 can be pruned to 38.5% sparsity with minimal quality loss, achieving a significant reduction in model size. We propose an optimal pruning configuration that prunes the text encoder to 47.5% and the diffusion generator to 35%. This configuration maintains image generation quality while substantially reducing computational requirements. In addition, our work uncovers intriguing questions about information encoding in text-to-image models: we observe that pruning beyond certain thresholds leads to sudden performance drops (unreadable images), suggesting that specific weights encode critical semantics information. This finding opens new avenues for future research in model compression, interoperability, and bias identification in text-to-image models. By providing crucial insights into the pruning behavior of text-to-image models, our study lays the groundwork for developing more efficient and accessible AI-driven image generation systems
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。