arXiv:2606.29013cs.CV2026-06

用冻结的LLM引导图像生成,无需多模态训练数据

Mural: Transferring LLM knowledge to image generation via Mixture-of-Transformers

论文配图:Mural: Transferring LLM knowledge to image generation via Mixture-of-Transformers
图 1 · 摘自论文原文
  • 将冻结的LLM通过MoT架构与扩散模型共享注意力
  • 在仅文本-图像对上训练,达成0.85/86.75/0.66基准性能
  • 实现跨语言生成、颜色引导构图等新能力

利用大语言模型(LLM)在文生图(T2I)合成中的能力是重要研究方向。本文探究在仅使用标准文本-图像对训练时,冻结的、具备推理能力的LLM知识是否可被有效利用。我们通过混合变压器(MoT)架构,在共享注意力机制下将冻结的LLM与基于扩散的图像生成器结合。实验涵盖两个关键问题:(1)训练过程中LLM的内在知识有多少仍可访问;(2)系统涌现出哪些新能力。在多个基准测试中,我们的模型在统一理解-生成系统中表现优异:GenEval得分为0.85,DPG-Bench为86.75,WISE为0.66(含推理时推理)。令人惊讶的是,系统展现出训练数据中不存在的涌现行为,包括跨语言图像生成、颜色引导构图、表情符号/ASCII场景构建以及基于世界知识的生成。结果表明,在标准文生图训练范式下,预训练的LLM知识可有效引导图像合成,无需交错的多模态信号或显式推理监督。这些发现为在资源受限的多模态学习中利用冻结模型能力开辟了新路径。

原文摘要 · Abstract (English)

Leveraging capabilities of large language models (LLMs) in text-to-image (T2I) synthesis is an important research direction. In this work we investigate whether the knowledge of a frozen LLM can be effectively utilized in T2I generation when trained exclusively on standard text-image pairs. We integrate a frozen, reasoning-capable LLM with a diffusion-based image generator via shared attention within the Mixture-of-Transformers (MoT) architecture. Our experiments span two critical questions: (1) what degree of the LLM's intrinsic knowledge remains accessible during T2I training, and (2) what novel capabilities emerge in the resulting system. Across established benchmarks, our models achieve strong performance among unified understanding-generation systems: 0.85 on GenEval, 86.75 on DPG-Bench, and 0.66 on WISE with inference-time reasoning, using only text-image data. Remarkably, we uncover emergent behaviors absent from training data, including cross-lingual image generation, color-guided composition, emoji / ASCII scene construction, and generation directed by world knowledge. These results demonstrate that pretrained LLM knowledge can guide image synthesis under standard text-to-image training paradigms, without interleaved multimodal signals or explicit reasoning supervision. Our findings open new avenues for harnessing frozen model capabilities in resource-constrained multimodal learning.

文生图LLM扩散模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。