通过去除高频词配对，提升视觉语言模型预训练效率与性能
Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency
- 基于文本中词汇频率筛选并剔除冗余图文对
- 使词频分布更均衡，显著提升下游任务表现
- 无需额外元数据，适用于各类视觉语言模型
我们提出基于词频的图文对剪枝方法（WFPP），一种提升视觉语言模型预训练效率的新方法。与MetaCLIP不同，WFPP不依赖元数据，而是根据文本内容选择要剪枝的图文对：剔除包含整个训练集中高频词的配对。该方法有效降低高频词的主导性，改善数据集中的词频分布均衡性，而均衡的词频分布已被证实有助于词嵌入模型的训练。在剪枝后的子集上完成预训练后，再对完整数据集进行一次额外的微调，可获得更优性能。实验表明，将WFPP应用于CLIP模型训练，可在多种下游任务上取得更好结果。同时，该方法通过减少训练样本数量，显著加快预训练速度。我们还分析了剪枝前后训练数据的词频分布变化，直观展示其平衡效果。希望本工作能推动研究者在预训练视觉语言模型时，关注训练数据中词频分布的合理性，不限于CLIP。
原文摘要 · Abstract (English)
We propose Word-Frequency-based Image-Text Pair Pruning (WFPP), a novel data pruning method that improves the efficiency of VLMs. Unlike MetaCLIP, our method does not need metadata for pruning, but selects text-image pairs to prune based on the content of the text. Specifically, WFPP prunes text-image pairs containing high-frequency words across the entire training dataset. The effect of WFPP is to reduce the dominance of frequent words. The result a better balanced word-frequency distribution in the dataset, which is known to improve the training of word embedding models. After pre-training on the pruned subset, we fine-tuned the model on the entire dataset for one additional epoch to achieve better performance. Our experiments demonstrate that applying WFPP when training a CLIP model improves performance on a wide range of downstream tasks. WFPP also provides the advantage of speeding up pre-training by using fewer samples. Additionally, we analyze the training data before and after pruning to visualize how WFPP changes the balance of word frequencies. We hope our work encourages researchers to consider the distribution of words in the training data when pre-training VLMs, not limited to CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。