开源视觉语言模型突破性能瓶颈，无需依赖闭源模型即可达到顶尖水平。
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- 自建高质量图文数据集，避免使用闭源模型生成训练数据。
- 720亿参数模型在多项评测中超越更大闭源模型，仅略逊于GPT-4o。
- 适合研究者复现、改进及构建完全开源的多模态系统。
当前最先进的视觉语言模型（VLM）仍为专有模型。现有最强的开源权重模型严重依赖闭源VLM生成的合成数据来提升性能，实质上是将闭源模型知识蒸馏到开源模型中。这导致社区缺乏从零构建高性能VLM的基础认知。本文提出Molmo系列VLM，其在开源权重与数据类别中处于领先地位。关键贡献是构建了名为PixMo的新数据集集合，包括用于预训练的高细节图像描述数据集、自由形式图像问答数据集，以及创新的二维指代标注数据集，全部通过人工采集，未使用外部VLM。模型成功依赖于精心设计的架构、优化的训练流程，尤其是新数据集的高质量。其最优720亿参数模型不仅超越同类开源模型，还在学术基准和大规模人类评估中优于更大的闭源模型（如Claude 3.5 Sonnet、Gemini 1.5 Pro/Flash），仅略逊于GPT-4o。模型权重、新数据集及源代码已公开：https://molmo.allenai.org/blog。
原文摘要 · Abstract (English)
Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。