超大二进制编码字典让多模态模型统一处理图像生成与理解。
UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model
- 用2^128级二进制码本统一图像编码，兼顾高保真重建与语义提取。
- 训练仅需330亿token，生成FID达1.38，优于现有方案。
- 适合需要跨分辨率、人脸和文字敏感场景的多模态应用。
统一多模态大语言模型（MLLM）需要一种视觉表示，同时满足高保真重建、复杂语义提取和生成适用性。然而，现有视觉分词器难以在单一框架内兼顾这些相互冲突的目标。本文提出UniWeTok，一个采用超大二进制码本（2^128）的统一离散分词器。通过引入预-后蒸馏和生成感知先验，增强离散标记的语义提取能力和生成先验。在架构上，提出卷积-注意力混合结构与SigLu激活函数，该函数不仅限制编码器输出，稳定语义蒸馏过程，还有效缓解了标记熵损失与承诺损失之间的优化冲突。进一步设计三阶段训练框架，提升对多种图像分辨率及人类面部、文本内容等感知敏感场景的适应性。在ImageNet上，UniWeTok实现顶尖图像生成性能（FID: 1.38 vs. REPA 1.42），训练计算量极低（训练令牌数：330亿 vs. REPA 2620亿）。在通用领域，其在多模态理解、图像生成（DPG得分：86.63 vs. FLUX.1 [Dev] 83.84）和编辑任务（GEdit总分：5.09 vs. OmniGen 5.06）中表现优异。代码与模型已开源，推动统一分词器与MLLM研究。
原文摘要 · Abstract (English)
Unified Multimodal Large Language Models (MLLMs) require a visual representation that simultaneously supports high-fidelity reconstruction, complex semantic extraction, and generative suitability. However, existing visual tokenizers typically struggle to satisfy these conflicting objectives within a single framework. In this paper, we introduce UniWeTok, a unified discrete tokenizer designed to bridge this gap using a massive binary codebook ($\mathit{2^{128}}$). For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens. In terms of model architecture, we propose a convolution-attention hybrid architecture with the SigLu activation function. SigLu activation not only bounds the encoder output and stabilizes the semantic distillation process but also effectively addresses the optimization conflict between token entropy loss and commitment loss. We further propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios, such as those involving human faces and textual content. On ImageNet, UniWeTok achieves state-of-the-art image generation performance (FID: UniWeTok 1.38 vs. REPA 1.42) while requiring a remarkably low training compute (Training Tokens: UniWeTok 33B vs. REPA 262B). On general-domain, UniWeTok demonstrates highly competitive capabilities across a broad range of tasks, including multimodal understanding, image generation (DPG Score: UniWeTok 86.63 vs. FLUX.1 [Dev] 83.84), and editing (GEdit Overall Score: UniWeTok 5.09 vs. OmniGen 5.06). We release code and models to facilitate community exploration of unified tokenizer and MLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。