用图文联合自回归训练大模型，性能超越现有对比学习方法。
Multimodal Autoregressive Pre-training of Large Vision Encoders
- 将自回归预训练扩展到图文联合任务，通过生成图像块和文本实现统一建模。
- AIMV2-3B在ImageNet-1k上达89.5%准确率，冻结主干仍表现优异。
- 适合多模态理解、视觉定位与分类等下游任务，尤其擅长跨模态对齐。
我们提出一种大规模视觉编码器的新型预训练方法。基于近期视觉模型自回归预训练的进展，将该框架拓展至多模态场景（图像与文本）。本文介绍的AIMV2是一系列通用型视觉编码器，具有流程简单、可扩展性强、下游任务表现突出等特点。其通过将视觉编码器与多模态解码器结合，自回归地生成原始图像块和文本标记。这些编码器不仅在多模态评估中表现卓越，在视觉基准任务如定位、指代消解和分类上也领先。值得注意的是，AIMV2-3B在冻结主干的前提下于ImageNet-1k上达到89.5%准确率；且在多种设置下持续优于当前最优对比学习模型（如CLIP、SigLIP）。
原文摘要 · Abstract (English)
We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framework to a multimodal setting, i.e., images and text. In this paper, we present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks. This is achieved by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in multimodal image understanding across diverse settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。