统一图像像素与文字的生成模型，提升细粒度视觉理解能力。
Unified Pix Token And Word Token Generative Language Model
- 将图像像素与文本统一为通用令牌，实现跨模态融合
- 小模型和少量数据下仍表现良好，符合规模扩展规律
- 适合需要精细图像理解的多模态生成任务
自视觉变压器（ViT）出现以来，广泛应用于生成式语言模型和生成式视觉模型。当前开源的先进多模态模型普遍采用通过CLIP或SigLIP方法训练的ViT作为视觉编码器骨干，以获得视觉理解能力。然而该方法在细节理解方面存在局限，例如难以识别图像中的小文本或数字。为此，本文提出一种新模型，将像素令牌（pix token）与词令牌（word token）统一到生成式语言模型中。该模型具备每个图像像素对应独立令牌嵌入、颜色折叠、全局条件注意力近似及图像无监督预训练等特性。我们使用该模型进行了图像无监督预训练实验，结果表明，即使在小型模型和有限训练数据条件下，性能依然出色。我们认为该模型符合规模定律：只要增加模型参数和训练数据，性能将持续提升。
原文摘要 · Abstract (English)
Since the emergence of Vision Transformer (ViT), it has been widely used in generative language model and generative visual model. Especially in the current state-of-art open source multimodal models, ViT obtained by CLIP or SigLIP method serves as the vision encoder backbone to help them acquire visual understanding capabilities. But this method leads to limitations in visual understanding for details, such as difficulty in recognizing small text or numbers in images. To address these issues, we propose a new model to unify pix token and word token into the generative language model. The new model also features with each pix of image having its own token embedding, color folding, global conditional attention approximation and image unsupervised pretraining. We conducted image unsupervised pretraining experiments using our new model to explore its potential. The experimental results show that it has good performance even in small model and with limited training data. We believe our model also conforms to the scaling law, as long as model parameters and training data increased, its performance will continue to improve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。