arXiv:2509.16197cs.CVcs.CL2025-09被引 16

统一视觉与文本生成,用混合分词器提升多模态模型性能

MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer

  • 采用混合图像分词器+共享编码器,理解与生成共用语义空间
  • 在文本密集型评测中达统一模型最佳表现,接近专用模型水平
  • 设计简洁可扩展,适合追求统一多模态能力的研究者使用

统一多模态大语言模型兼具视觉理解与生成能力,但开源模型常面临性能权衡。本文提出Manzano,通过混合图像分词器与精心设计的训练方案,显著缓解该矛盾。单一共享视觉编码器连接两个轻量适配器:一个输出连续嵌入用于图文理解,另一个生成离散标记用于文生图生成,二者均在共同语义空间中。统一自回归语言模型同时预测文本与图像标记,再由辅助扩散解码器将图像标记转为像素。结合理解与生成数据的统一训练策略,实现两种能力的可扩展联合学习。实验表明,Manzano在统一模型中达到最先进水平,尤其在文本密集型评测中表现优异,且任务间冲突极小,模型规模扩大时性能持续提升,验证了混合分词器设计的有效性。

原文摘要 · Abstract (English)

Unified multimodal Large Language Models (LLMs) that can both understand and generate visual content hold immense potential. However, existing open-source models often suffer from a performance trade-off between these capabilities. We present Manzano, a simple and scalable unified framework that substantially reduces this tension by coupling a hybrid image tokenizer with a well-curated training recipe. A single shared vision encoder feeds two lightweight adapters that produce continuous embeddings for image-to-text understanding and discrete tokens for text-to-image generation within a common semantic space. A unified autoregressive LLM predicts high-level semantics in the form of text and image tokens, with an auxiliary diffusion decoder subsequently translating the image tokens into pixels. The architecture, together with a unified training recipe over understanding and generation data, enables scalable joint learning of both capabilities. Manzano achieves state-of-the-art results among unified models, and is competitive with specialist models, particularly on text-rich evaluation. Our studies show minimal task conflicts and consistent gains from scaling model size, validating our design choice of a hybrid tokenizer.

多模态统一模型生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。