解释了为何用模型筛选数据能提升多模态学习效果
Understanding the Gain from Data Filtering in Multimodal Contrastive Learning
- 用预训练模型为网页数据打质量分，筛选出匹配正确的图文对
- 理论证明：过滤后误差比未过滤时更低，尤其当正确数据占比高时
- 适合研究多模态训练数据清洗或对比学习机制的学者
现代多模态表示学习依赖互联网规模的数据集。由于大量原始网络数据质量较低，数据清洗已成为训练流程中的关键步骤。基于训练模型（即教师模型）的过滤方法成为有效方案，利用预训练模型计算数据质量分数。为解释该方法的实证成功，本文在标准双模态数据生成模型下分析了过滤对比学习的表现。设总样本数为 $n$，其中模态正确匹配的比例为 $ηig(∈(0,1]ig)$，通过线性对比学习设置，证明了数据过滤的可证明优势：(i) 无过滤时误差上下界均为 $ rac{1}{η oot{}{n}}$；(ii) 教师模型过滤后，在 $η$ 较大时误差上界为 $ rac{1}{ oot{}{ηn}}$，在 $η$ 较小时为 $ rac{1}{ oot{}{n}}$。
原文摘要 · Abstract (English)
The success of modern multimodal representation learning relies on internet-scale datasets. Due to the low quality of a large fraction of raw web data, data curation has become a critical step in the training pipeline. Filtering using a trained model (i.e., teacher-based filtering) has emerged as a successful solution, leveraging a pre-trained model to compute quality scores. To explain the empirical success of teacher-based filtering, we characterize the performance of filtered contrastive learning under the standard bimodal data generation model. Denoting $η\in(0,1]$ as the fraction of data with correctly matched modalities among $n$ paired samples, we utilize a linear contrastive learning setup to show a provable benefit of data filtering: $(i)$ the error without filtering is upper and lower bounded by $\frac{1}{η\sqrt{n}}$, and $(ii)$ the error with teacher-based filtering is upper bounded by $\frac{1}{\sqrt{ηn}}$ in the large $η$ regime, and by $\frac{1}{\sqrt{n}}$ in the small $η$ regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。