让预训练视觉模型同时具备理解和生成图像能力
Growing Visual Generative Capacity for Pre-Trained MLLMs
- 用混合变换器架构在原有模型上加生成能力
- 语义与像素级令牌结合,序列长度仅增7.9%
- 理解生成都强,训练数据和时间更少
多模态大语言模型将语言模型的成功拓展至视觉理解,近期研究致力于构建兼具理解与生成能力的统一模型。然而,构建此类模型仍具挑战:混合方法结合连续嵌入与扩散或流模型目标,虽生成高质量图像但破坏自回归范式;纯自回归方法虽统一文本与图像预测,但在语义对齐与像素级保真度间存在权衡。本文提出Bridge,一种纯自回归的统一多模态大模型,通过混合变换器架构,为预训练视觉理解模型注入生成能力,实现单一下一步词预测框架下的图像理解与生成。为提升生成保真度,提出语义到像素的离散表示,融合紧凑语义令牌与细粒度像素令牌,在仅增加7.9%序列长度的情况下,实现强语言对齐与精确视觉细节描述。跨多种多模态基准的大量实验表明,Bridge在理解和生成任务中均达到竞争性或更优表现,且所需训练数据和训练时间显著低于先前统一多模态大模型。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) extend the success of language models to visual understanding, and recent efforts have sought to build unified MLLMs that support both understanding and generation. However, constructing such models remains challenging: hybrid approaches combine continuous embeddings with diffusion or flow-based objectives, producing high-quality images but breaking the autoregressive paradigm, while pure autoregressive approaches unify text and image prediction over discrete visual tokens but often face trade-offs between semantic alignment and pixel-level fidelity. In this work, we present Bridge, a pure autoregressive unified MLLM that augments pre-trained visual understanding models with generative ability through a Mixture-of-Transformers architecture, enabling both image understanding and generation within a single next-token prediction framework. To further improve visual generation fidelity, we propose a semantic-to-pixel discrete representation that integrates compact semantic tokens with fine-grained pixel tokens, achieving strong language alignment and precise description of visual details with only a 7.9% increase in sequence length. Extensive experiments across diverse multimodal benchmarks demonstrate that Bridge achieves competitive or superior results in both understanding and generation benchmarks, while requiring less training data and reduced training time compared to prior unified MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。