用文本对齐的离散编码统一视觉理解与生成
Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
- 通过文本对齐的代码本将图像转为离散标记
- 多模态模型Tar在多个任务上超越现有方法
- 支持快速自回归与扩散生成，适合多场景应用
本文提出一种多模态框架，旨在通过共享的离散语义表示统一视觉理解与生成。核心是文本对齐分词器（TA-Tok），利用大语言模型（LLM）词汇投影生成的代码本，将图像转换为离散标记。通过将视觉与文本融合至扩展词汇空间的统一表示中，我们的多模态大模型Tar实现了跨模态输入输出，无需模态专用设计。同时，引入尺度自适应编码与解码以平衡效率与细节，并设计生成式解码器实现高保真视觉输出。针对不同解码需求，采用两种互补解码器：快速自回归模型与基于扩散的模型。通过先进预训练任务增强模态融合，实验表明该方法在多个基准测试中表现优异，达到更快收敛速度与更高训练效率。代码、模型与数据已公开于https://tar.csuhan.com
原文摘要 · Abstract (English)
This paper presents a multimodal framework that attempts to unify visual understanding and generation within a shared discrete semantic representation. At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into discrete tokens using a text-aligned codebook projected from a large language model's (LLM) vocabulary. By integrating vision and text into a unified space with an expanded vocabulary, our multimodal LLM, Tar, enables cross-modal input and output through a shared interface, without the need for modality-specific designs. Additionally, we propose scale-adaptive encoding and decoding to balance efficiency and visual detail, along with a generative de-tokenizer to produce high-fidelity visual outputs. To address diverse decoding needs, we utilize two complementary de-tokenizers: a fast autoregressive model and a diffusion-based model. To enhance modality fusion, we investigate advanced pre-training tasks, demonstrating improvements in both visual understanding and generation. Experiments across benchmarks show that Tar matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency. Code, models, and data are available at https://tar.csuhan.com
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。