用字节对编码方法对图像进行分词，提升多模态模型理解能力。
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
- 将文本的字节对编码思想迁移至图像，直接融合结构先验信息
- 在数据有限时仍显著提升多模态模型的跨模态理解能力
- 适合追求高效、可扩展多模态基础模型的研究者与开发者
多模态大语言模型在融合视觉与文本信息方面取得显著进展，但仍面临模态对齐困难的问题。本文提出一种新型图像分词器，将字节对编码（BPE）原理应用于视觉数据。不同于依赖独立视觉编码器的传统方法，该方法直接将结构先验信息融入图像令牌，模仿纯文本大模型的成功分词策略。通过理论分析与大量实验，我们证明该BPE图像分词器能显著增强多模态大模型的跨模态理解能力，即便在训练数据有限的情况下亦然。基于此方法，我们构建了Being-VL-0模型，在多个基准测试中表现优异，展现出良好可扩展性，有望为更高效、更强大的多模态基础模型铺平道路。
原文摘要 · Abstract (English)
Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。