用视觉知识蒸馏让文本编码器缩小50倍，质量不降反而更省显存。
Scaling Down Text Encoders of Text-to-Image Diffusion Models
- 基于图像质量、语义理解与文字渲染构建数据集，进行视觉知识蒸馏。
- 蒸馏后的T5-base模型仅占T5-XXL的1/50，生成图像质量相当。
- 适合想在低配设备上跑高质量文生图模型的研究者和开发者。
扩散模型中的文本编码器已从CLIP演进至T5-XXL，显著提升对复杂提示的理解能力，但参数量急剧增加。尽管T5系列在包含大量非视觉内容的C4语料库上训练，现有扩散模型仍无法响应非视觉提示，表明存在表征冗余。为此，本文采用基于视觉的知识蒸馏方法，构建符合图像质量、语义理解与文字渲染三标准的数据集，训练一系列精简版T5编码器。实验结果表明，蒸馏后的T5-base模型在生成图像质量上可媲美T5-XXL，但模型尺寸仅为后者的1/50。该压缩显著降低如FLUX与SD3等先进模型的显存需求，使高质量文生图生成更易部署于资源受限环境。
原文摘要 · Abstract (English)
Text encoders in diffusion models have rapidly evolved, transitioning from CLIP to T5-XXL. Although this evolution has significantly enhanced the models' ability to understand complex prompts and generate text, it also leads to a substantial increase in the number of parameters. Despite T5 series encoders being trained on the C4 natural language corpus, which includes a significant amount of non-visual data, diffusion models with T5 encoder do not respond to those non-visual prompts, indicating redundancy in representational power. Therefore, it raises an important question: "Do we really need such a large text encoder?" In pursuit of an answer, we employ vision-based knowledge distillation to train a series of T5 encoder models. To fully inherit its capabilities, we constructed our dataset based on three criteria: image quality, semantic understanding, and text-rendering. Our results demonstrate the scaling down pattern that the distilled T5-base model can generate images of comparable quality to those produced by T5-XXL, while being 50 times smaller in size. This reduction in model size significantly lowers the GPU requirements for running state-of-the-art models such as FLUX and SD3, making high-quality text-to-image generation more accessible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。