arXiv:2412.04332cs.CV2024-12IJCV被引 66

用单一大模型实现图文生成与理解,性能随规模提升而增强。

Liquid: Language Models are Scalable and Unified Multi-modal Generators

  • 将图像转为离散代码,与文本共用同一特征空间。
  • 模型越大,图文联合训练的性能下降越小,突破以往瓶颈。
  • 仅需1/100训练成本,超越多模态模型Chameleon和Stable Diffusion。

我们提出Liquid,一种自回归生成范式,通过将图像编码为离散代码,并将其嵌入与文本令牌共享的统一特征空间中,实现视觉理解与生成的无缝融合。与以往多模态大语言模型(MLLM)不同,Liquid仅使用单一大型语言模型(LLM),无需外部预训练视觉嵌入(如CLIP)。首次揭示了随着模型规模增大,图文联合训练带来的性能下降会逐渐减弱的缩放定律。统一的令牌空间使视觉生成与理解任务相互促进,有效消除早期模型中的典型干扰。我们证明现有LLM可作为Liquid的强大基础,节省100倍训练成本,同时在多模态能力上超越Chameleon,语言性能媲美主流模型如LLAMA2。Liquid在视觉-语言及纯文本任务中均表现优异,其在MJHQ-30K数据集上的FID达到5.47,优于SD v2.1和SD-XL。本工作表明Qwen2.5和GEMMA2等模型具备强大的多模态生成能力,为视觉-语言理解与生成提供可扩展解决方案。代码与模型将开源于https://github.com/FoundationVision/Liquid。

原文摘要 · Abstract (English)

We present Liquid, an auto-regressive generation paradigm that seamlessly integrates visual comprehension and generation by tokenizing images into discrete codes and learning these code embeddings alongside text tokens within a shared feature space for both vision and language. Unlike previous multimodal large language model (MLLM), Liquid achieves this integration using a single large language model (LLM), eliminating the need for external pretrained visual embeddings such as CLIP. For the first time, Liquid uncovers a scaling law that performance drop unavoidably brought by the unified training of visual and language tasks diminishes as the model size increases. Furthermore, the unified token space enables visual generation and comprehension tasks to mutually enhance each other, effectively removing the typical interference seen in earlier models. We show that existing LLMs can serve as strong foundations for Liquid, saving 100x in training costs while outperforming Chameleon in multimodal capabilities and maintaining language performance comparable to mainstream LLMs like LLAMA2. Liquid also outperforms models like SD v2.1 and SD-XL (FID of 5.47 on MJHQ-30K), excelling in both vision-language and text-only tasks. This work demonstrates that LLMs such as Qwen2.5 and GEMMA2 are powerful multimodal generators, offering a scalable solution for enhancing both vision-language understanding and generation. The code and models will be released at https://github.com/FoundationVision/Liquid.

多模态生成模型大模型统一空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。