用现成模型筛选多模态数据,省去训练成本还提升效果
DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models

- 直接用未见过目标数据的预训练模型评估图文质量与对齐度
- 通过加权采样保留长尾多样性,使小模型性能媲美全量数据训练
- 无需微调即可高效过滤噪声,适合大规模多模态数据处理
高质量且多样化的多模态数据对提升视觉语言模型(VLMs)至关重要,但现有数据集常包含噪声、冗余和对齐不佳的样本。为解决此问题,通常采用数据过滤提升学习效率与性能,但传统方法需在待筛选数据上训练过滤模型,带来额外计算开销。为此,本文提出DOSE,探索是否可使用从未见过目标数据的现成预训练模型,在不进行任务特定训练的情况下,有效筛选大型强模型的训练样本。实验表明,这些模型即使未微调,也能准确评估文本质量与图像-文本对齐程度。基于此,构建联合质量-对齐分布,采用自适应加权采样策略,选取信息量大且保持长尾多样性样本。在多个标准视觉问答(VQA)与数学推理基准上,使用DOSE筛选数据训练的模型表现达到甚至超过使用完整数据集训练的模型。大量实验验证了该方法的有效性、高效性与可扩展性。
原文摘要 · Abstract (English)
High-quality and diverse multimodal data are essential for improving vision-language models (VLMs), yet existing datasets often contain noisy, redundant, and poorly aligned samples. To address these problems, data filtering is commonly used to enhance the efficiency and performance of multimodal learning, but it introduces extra computational cost because filtering models are usually trained on the same data they are meant to screen. To reduce this cost, we study DOSE, which explores whether off-the-shelf pretrained models that have never seen the target data can be used to select training samples for larger and stronger multimodal models without any task-specific training. Even without fine-tuning, these models can effectively assess text quality and image-text alignment to guide data selection. Based on this, we build a joint quality-alignment distribution and apply adaptive weighted sampling to select informative samples while maintaining long-tail diversity. This approach enhances data diversity, enabling models trained on DOSE-filtered data to match or surpass those trained on the full dataset on standard VQA and math benchmarks. Extensive experiments demonstrate its effectiveness, efficiency, and scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。