用下游任务反馈自动筛选视觉语言数据，提升CLIP预训练效果
Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
- 根据下游任务梯度信号训练评分模型，判断每条数据价值
- 在DataComp中达40.1% ImageNet零样本准确率，比此前结果高2%
- 适合做大规模视觉语言数据清洗的研究者和工程师
我们提出Filter Like You Test（FLYT），一种用于构建大规模视觉语言数据集的算法，可学习每个数据点作为预训练样本的有效性。FLYT训练一个评分模型，利用下游任务训练集的梯度信号来加权每个样本的特征。基于FLYT，我们实现了Mixing-FLYT（M-FLYT），将不同评分方法生成的样本分数作为特征，学习融合为单一分数。FLYT自然产生样本的分布，我们通过软上限采样（SCS）策略，利用每个样本的概率进行过滤，通过重复惩罚防止过度采样。使用这些方法，在DataComp中等规模过滤基准上，实现40.1%的ImageNet零样本准确率，相比此前所有结果提升2个百分点，相比仅使用公开资源的方法提升5.5个百分点。该方法在38个DataComp评估任务上的平均准确率达37.7%，优于以往仅使用公开资源的方法0.4个百分点。
原文摘要 · Abstract (English)
We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that learns the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks training sets. Based on FLYT, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces a distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, we achieve 40.1% ImageNet zero-shot accuracy on the DataComp medium scale filtering benchmark, a 2% absolute accuracy increase over all previous results and a 5.5% increase over results that - like us - use only public resources. Our approach also yields 37.7\% on the average of 38 DataComp evaluation tasks, outperforming previous public-resource approaches by 0.4\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。