arXiv:2511.20643cs.CVcs.LG2025-11被引 1

通过动态选择图像文本对,让多模态模型学得更全面、更聚焦。

Concept-Aware Batch Sampling Improves Language-Image Pretraining

  • 按概念分布实时构建训练批次,灵活调整数据组成。
  • 在28个基准上提升CLIP/SigLIP模型性能,最高增益达12.3%。
  • 适合需要定制化数据训练的科研与工业应用。

视觉语言模型该用什么数据训练?现有方法多依赖静态数据筛选,且忽略概念差异,导致引入额外偏差。本文提出DataConcept,一个包含1.28亿条网络爬取的图文对的数据集,细粒度标注了概念构成。基于此,我们设计Concept-Aware Batch Sampling(CABS),一种在线动态批处理采样框架,可灵活根据目标概念分布构建批次。提出两种变体:(i) 多样性最大化(CABS-DM),提升概念覆盖广度;(ii) 频率最大化(CABS-FM),增强高频物体出现次数。在28个基准上的实验证明,CABS显著提升CLIP/SigLIP模型表现,取得优异性能。CABS为开源提供强大替代方案,使用户可根据下游任务自定义概念分布。

原文摘要 · Abstract (English)

What data should a vision-language model be trained on? To answer this question, many data curation efforts center on the quality of a dataset. However, most of these existing methods are (i) offline, i.e. they produce a static dataset from a set of predetermined filtering criteria, and (ii) concept-agnostic, i.e. they use model-based filters which induce additional data biases. In this work, we go beyond such offline, concept-agnostic methods and advocate for more flexible, task-adaptive online concept-based curation. Our first contribution is DataConcept, a collection of 128M web-crawled image-text pairs annotated with fine-grained details about their concept composition. Building on DataConcept, we introduce Concept-Aware Batch Sampling (CABS), a simple yet effective batch sampling framework that flexibly constructs batches on-the-fly based on specific target distributions. We propose two variants: (i) Diversity Maximization (CABS-DM) to curate batches with a broad coverage of available concepts, and (ii) Frequency Maximization (CABS-FM) to curate batches with high object multiplicity. Through extensive evaluations across 28 benchmarks, we demonstrate that our CABS method significantly benefits CLIP/SigLIP model classes and yields highly performant models. Overall, CABS represents a strong open-source alternative to proprietary online data curation algorithms, enabling practitioners to define custom concept distributions that optimize for specific downstream tasks.

多模态数据采样预训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。