用少量成本实现110种语言的图像生成，效果接近英语。
MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
- 用互联网噪声数据训练轻量语言适配器，替代全模型微调。
- 110种语言生成质量接近英语，相似度达39.61（英语为39.57）。
- 兼容LoRA等工具，适合快速部署多语言生成应用。
本文提出一种低成本的多语言图像生成框架。与依赖高质量多语言图文数据的模型不同，我们发现利用广泛存在的、带有噪声的英文图文对预训练文本编码器，能显著提升文本到图像生成的数据效率。基于此，提出MuLan——一个参数少于2000万的轻量级多语言适配器，与冻结的文本编码器和图像扩散模型联合训练。相比以往多语言图像生成模型，该框架具备三大优势：（1）成本低：仅需现成的英文数据和通用多语言文本编码器，极大降低训练成本；（2）性能高：在超过110种语言上表现优异，平均CLIP相似度达39.61，接近英语的39.57；（3）适用广：可无缝集成至LoRA、LCM、ControlNet及IP-Adapter等社区工具，拓展应用场景。
原文摘要 · Abstract (English)
In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages.Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。