用单个Transformer实现视觉语言模型，性能媲美复杂架构。
The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
- 仅用一个Transformer融合图像像素与语言解码，无独立视觉编码器。
- 在数据和模型规模扩大时，性能接近模块化模型，且跨模态流动模式不同。
- 无需预训练ViT，可直接用于视觉任务，如语义分割表现优异。
本文提出SAIL，一种基于单一Transformer的统一多模态大语言模型，将原始像素编码与语言解码集成于同一架构中。不同于依赖预训练视觉变换器（ViT）的模块化多模态模型，SAIL省去独立视觉编码器，采用混合注意力机制与多模态位置编码，更贴合视觉与文本模态特性。通过系统比较SAIL与模块化模型在可扩展性、跨模态信息流模式及视觉表征能力方面的差异，发现其在扩大训练数据与模型规模后，性能可达模块化模型水平。移除预训练ViT显著提升可扩展性，并导致截然不同的跨模态信息流动模式。此外，SAIL在语义分割等视觉任务上表现优异，结果与ViT-22B相当。代码与模型已公开于https://github.com/bytedance/SAIL。
原文摘要 · Abstract (English)
This paper introduces SAIL, a single transformer unified multimodal large language model (MLLM) that integrates raw pixel encoding and language decoding within a singular architecture. Unlike existing modular MLLMs, which rely on a pre-trained vision transformer (ViT), SAIL eliminates the need for a separate vision encoder, presenting a more minimalist architecture design. Instead of introducing novel architectural components, SAIL adapts mix-attention mechanisms and multimodal positional encodings to better align with the distinct characteristics of visual and textual modalities. We systematically compare SAIL's properties-including scalability, cross-modal information flow patterns, and visual representation capabilities-with those of modular MLLMs. By scaling both training data and model size, SAIL achieves performance comparable to modular MLLMs. Notably, the removal of pretrained ViT components enhances SAIL's scalability and results in significantly different cross-modal information flow patterns. Moreover, SAIL demonstrates strong visual representation capabilities, achieving results on par with ViT-22B in vision tasks such as semantic segmentation. Code and models are available at https://github.com/bytedance/SAIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。