根据词频设计文本掩码，提升视觉语言模型预训练效率
Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training
- 按词频高低选择性掩码文本，增强模型对高频词的感知
- 在低输入令牌数下，新方法比语法掩码提升显著，最高达7.2%准确率
- 适用于资源受限场景，尤其适合长训练周期的模型优化
视觉语言模型（VLM）若能减少训练数据量，则可更高效训练。近期研究发现，在训练中对文本进行不同策略的掩码（如截断、随机掩码、块掩码和语法掩码）有益，其中语法掩码表现最佳。本文分析了不同文本掩码策略对训练数据中词频分布的影响，发现该影响与模型性能密切相关。据此提出对比语言图像预训练的词频掩码方法（CLIPF），直接利用词频信息进行掩码。大量实验表明，CLIPF在输入令牌数量减少时优于语法掩码及其他现有方法。此外，当训练轮次足够多时，不仅CLIPF，其他掩码策略也超越语法掩码，这一发现对VLM训练中掩码方法的选择具有实际意义。代码已公开。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) can be trained more efficiently if training sets can be reduced in size. Recent work has shown the benefits of masking text during VLM training using a variety of strategies (truncation, random masking, block masking and syntax masking) and has reported syntax masking as the top performer. In this paper, we analyze the impact of different text masking strategies on the word frequency in the training data, and show that this impact is connected to model success. This finding motivates Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF), our proposed masking approach, which directly leverages word frequency. Extensive experiments demonstrate the advantages of CLIPF over syntax masking and other existing approaches, particularly when the number of input tokens decreases. We show that not only CLIPF, but also other existing masking strategies, outperform syntax masking when enough epochs are used during training, a finding of practical importance for selecting a text masking method for VLM training. Our code is available online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。