构建视觉语言模型训练数据基准,发现混合指令数据比单纯筛选更有效。
DataComp-VLM: Improved Open Datasets for Vision-Language Models

- 设计可控实验框架,测试数据筛选、混合、格式等策略对模型影响。
- 80亿参数模型用2000亿令牌训练,核心任务准确率达63.6%,提升5.4个百分点。
- 开源160个数据集和完整工具链,适合研究数据优化与模型训练的团队使用。
构建高性能视觉语言模型(VLMs)需要精心设计大规模训练数据集,但社区缺乏系统性评估数据筛选策略的基准。本文提出针对VLM的数据中心实验基准DataComp for VLMs(DCVLM),涵盖四种数据类型:图像-文本对、多模态文档、纯文本与指令微调数据,共整合160个数据集,形成6万亿多模态标记的语料库。该基准支持在10亿至80亿参数模型及62.5亿至2000亿标记预算下,测试数据筛选、混合、格式化、采样等策略。模型在9个领域共计52个下游任务上进行评估。大量实验表明,数据混合比过滤更为关键:以指令为主的混合数据在更大规模下表现更好,增益随规模扩大而增加。由此构建的基准数据集DCVLM-Baseline,使80亿参数模型在33项核心任务上达到63.6%准确率,相比当前最佳开源数据集FineVision提升5.4个百分点。DCVLM及所有配套资源将公开发布于https://www.datacomp.ai/dcvlm/。
原文摘要 · Abstract (English)
Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。