用大模型提升文生图效果，零样本支持多语言生成
LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation
- 用大模型优化文本表征，分层优化提示词语义
- 轻量适配器实现图文特征高效对齐，训练快
- 无需额外训练即可跨语言生成，适合多语言应用
本文提出LDGen，一种将大语言模型（LLM）融入现有文生图扩散模型的新方法，显著降低计算开销。传统文本编码器如CLIP和T5在多语言处理上存在局限，影响跨语言图像生成效果。我们利用LLM的强语义理解能力，通过分层提示优化与人类指令技术提取精准语义信息，并引入轻量级适配器与跨模态精炼器，实现LLM与图像特征间的高效对齐与交互。实验表明，该方法在提示遵循度与图像美学质量上均优于基线模型，且可实现零样本多语言图像生成。项目页面：https://zrealli.github.io/LDGen。
原文摘要 · Abstract (English)
In this paper, we introduce LDGen, a novel method for integrating large language models (LLMs) into existing text-to-image diffusion models while minimizing computational demands. Traditional text encoders, such as CLIP and T5, exhibit limitations in multilingual processing, hindering image generation across diverse languages. We address these challenges by leveraging the advanced capabilities of LLMs. Our approach employs a language representation strategy that applies hierarchical caption optimization and human instruction techniques to derive precise semantic information,. Subsequently, we incorporate a lightweight adapter and a cross-modal refiner to facilitate efficient feature alignment and interaction between LLMs and image features. LDGen reduces training time and enables zero-shot multilingual image generation. Experimental results indicate that our method surpasses baseline models in both prompt adherence and image aesthetic quality, while seamlessly supporting multiple languages. Project page: https://zrealli.github.io/LDGen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。