用可学习查询实现图文大模型与扩散模型的高效互传
Transfer between Modalities with MetaQueries
- 设计可学习查询作为图文模型与扩散模型间的桥梁
- 仅需图像-文本配对数据，冻结主干模型仍能生成高质量图像
- 支持图像编辑和主题驱动生成，适配指令微调
统一多模态模型旨在融合理解（文本输出）与生成（像素输出），但将不同模态对齐于单一架构常需复杂的训练策略和精细的数据平衡。本文提出MetaQueries，一组可学习查询，作为自回归多模态大模型（MLLM）与扩散模型之间的高效接口。该方法将MLLM的潜在表示连接至扩散解码器，利用MLLM的深层理解与推理能力实现知识增强的图像生成。本方法简化了训练流程，仅需图像-文本配对数据及标准扩散目标。尤为关键的是，即使保持MLLM主干冻结，该迁移仍具有效性，从而在保留其领先多模态理解能力的同时，实现强大的生成性能。此外，该方法具备灵活性，可轻松进行指令微调，适用于图像编辑、主题驱动生成等高级应用。
原文摘要 · Abstract (English)
Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。