通过跳过与复用文本编码层，显著降低图像生成模型内存占用。
Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
- 针对文本编码器冗余性，设计跳过与复用策略以减少内存消耗。
- 在高稀疏度下仍保持与原模型相当的图像质量，优于现有剪枝方法。
- 适合资源受限场景下的高效文生图应用，尤其关注内存优化者。
大规模文本编码器在文生图扩散模型中表现出卓越性能，能生成高质量图像。尽管其单次前向传播对推理时间与浮点运算量贡献极小，却占用了高达八倍于去噪模块的内存。为此，我们提出专为文生图任务设计的跳过与复用层（Skrr）策略，通过有选择地跳过或复用变压器块中的部分层，有效利用编码器内在冗余性，在不损失性能的前提下大幅降低内存开销。大量实验表明，即使在高稀疏度下，Skrr仍能保持与原始模型相当的图像质量，超越现有逐块剪枝方法。此外，该方法在多种评估指标（包括FID、CLIP、DreamSim和GenEval）上均实现领先水平的内存效率与性能平衡。
原文摘要 · Abstract (English)
Large-scale text encoders in text-to-image (T2I) diffusion models have demonstrated exceptional performance in generating high-quality images from textual prompts. Unlike denoising modules that rely on multiple iterative steps, text encoders require only a single forward pass to produce text embeddings. However, despite their minimal contribution to total inference time and floating-point operations (FLOPs), text encoders demand significantly higher memory usage, up to eight times more than denoising modules. To address this inefficiency, we propose Skip and Re-use layers (Skrr), a simple yet effective pruning strategy specifically designed for text encoders in T2I diffusion models. Skrr exploits the inherent redundancy in transformer blocks by selectively skipping or reusing certain layers in a manner tailored for T2I tasks, thereby reducing memory consumption without compromising performance. Extensive experiments demonstrate that Skrr maintains image quality comparable to the original model even under high sparsity levels, outperforming existing blockwise pruning methods. Furthermore, Skrr achieves state-of-the-art memory efficiency while preserving performance across multiple evaluation metrics, including the FID, CLIP, DreamSim, and GenEval scores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。