用现代大模型做图像生成的文本编码器，效果优于传统T5。
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
- 改用Decoder-only大模型提取文本嵌入，取代旧式T5和CLIP。
- 多层归一化平均嵌入使复杂提示对齐更准，性能超越基线T5。
- 适合关注视觉语言理解、图文生成的科研与工程人员。
文本到图像生成与大语言模型（LLM）均取得显著进展。然而，多数文本到图像模型仍采用较陈旧的T5和CLIP作为文本编码器。本文研究将现代Decoder-only LLMs作为文本到图像扩散模型的文本编码器的有效性。我们构建了标准化训练与评估流程，以隔离并评估不同文本嵌入的影响。共训练27个文本到图像模型，使用12种不同文本编码器，分析影响生成效果的关键因素，包括嵌入提取方式、不同LLM变体及模型规模。实验表明，使用最后一层嵌入作为条件输入的效果较差；通过探索多层嵌入，发现跨所有层进行层归一化平均可显著提升与复杂提示的对齐能力。采用该策略的大多数LLM表现优于基线T5模型，在高级视觉语言推理任务中展现更强能力。
原文摘要 · Abstract (English)
Both text-to-image generation and large language models (LLMs) have made significant advancements. However, many text-to-image models still employ the somewhat outdated T5 and CLIP as their text encoders. In this work, we investigate the effectiveness of using modern decoder-only LLMs as text encoders for text-to-image diffusion models. We build a standardized training and evaluation pipeline that allows us to isolate and evaluate the effect of different text embeddings. We train a total of 27 text-to-image models with 12 different text encoders to analyze the critical aspects of LLMs that could impact text-to-image generation, including the approaches to extract embeddings, different LLMs variants, and model sizes. Our experiments reveal that the de facto way of using last-layer embeddings as conditioning leads to inferior performance. Instead, we explore embeddings from various layers and find that using layer-normalized averaging across all layers significantly improves alignment with complex prompts. Most LLMs with this conditioning outperform the baseline T5 model, showing enhanced performance in advanced visio-linguistic reasoning skills.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。