通过高质量数据构建，实现大模型高效训练并达到顶尖性能。
Scalable Vision Language Model Training via High Quality Data Curation
- 构建百亿级高质量图文数据集SAIL-Caption，提升数据可信度。
- 预训练使用6550亿token，2B模型性能随数据量呈对数增长。
- 分阶段用复杂度递增的数据微调，显著超越单阶段训练。
本文提出SAIL-VL（ScAlable Vision Language Model TraIning via High QuaLity Data Curation），一个开源视觉语言模型系列，在20亿和80亿参数规模下达到当前最优性能。其优势源于三项关键改进：(1) 可扩展的高质量视觉理解数据构建：设计数据构建流程，实现百亿级高质量图文重标注，生成的数据集SAIL-Caption在开源中质量最高；(2) 基于高质量数据的大规模预训练：将预训练预算扩大至6550亿词元，证明即使20亿参数模型也受益于数据量增加，性能遵循对数数据缩放规律；(3) 数据数量与复杂度双重扩展的可扩展微调：构建高质量指令微调数据集，采用逐步提升复杂度的训练策略，显著优于传统单阶段训练。SAIL-VL系列在18个主流视觉语言模型评测中平均得分最高，20亿模型在OpenCompass 2024榜单中领先同规模模型，展现出卓越的视觉理解能力。模型已发布于HuggingFace（https://huggingface.co/BytedanceDouyinContent）。
原文摘要 · Abstract (English)
In this paper, we introduce SAIL-VL (ScAlable Vision Language Model TraIning via High QuaLity Data Curation), an open-source vision language model (VLM) series achieving state-of-the-art (SOTA) performance in 2B and 8B parameters. The following three key improvements contribute to SAIL-VL's leading performance: (1) Scalable high-quality visual understanding data construction: We implement a data construction pipeline to enable hundred-million-scale high-quality recaption data annotation. The resulted dataset SAIL-Caption is validated to be of the highest data quality compared with opensource datasets. (2) Scalable Pretraining with High-Quality Visual Understanding Data: We scale SAIL-VL's pretraining budget up to 655B tokens and show that even a 2B VLM benefits from scaled up training data sizes, exhibiting logarithmic data size scaling laws in benchmark performance. (3) Scalable SFT via data quantity and complexity scaling: We curate a high-quality SFT dataset collection with leading data quantity scaling effectiveness and demonstrate that training with progressively higher-complexity data surpasses baseline one-stage training by a large margin. SAIL-VL series models achieve the highest average score in 18 widely used VLM benchmarks in our evaluation, with the 2B model takes the top position over VLMs of comparable sizes on OpenCompass 2024 (https://rank.opencompass.org.cn/leaderboard-multimodal), demonstrating robust visual comprehension abilities. SAIL-VL series models are released at HuggingFace (https://huggingface.co/BytedanceDouyinContent).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。