用模型权重模拟梯度对齐，高效筛选高质量数据
Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights
- 通过样本梯度与预训练模型方向的对齐度评估数据价值
- 减少20.7%训练步数实现相同CLIP性能，节省470万样本
- 无需验证集、计算量小，适合大规模数据清洗场景
大规模网络爬取数据集包含噪声、偏差和无关信息，需进行数据筛选。现有方法依赖人工设计规则、下游数据集或昂贵的影响计算，限制可扩展性并引入数据依赖。为此，我们提出Mimic Score，一种基于几何结构的数据质量度量，通过测量样本梯度与预训练参考模型诱导目标方向的对齐程度来评估数据效用。该方法利用现成模型权重，无需验证集，计算开销极低。基于此，我们提出Grad-Mimic框架：在线重加权加速训练，离线聚合构建高效数据过滤器。实证表明，使用Mimic Score指导训练可提升数据效率，加快收敛，在六个图像数据集上表现一致提升；使CLIP模型在减少20.7%训练步数下保持性能。此外，基于Mimic Score的过滤器增强现有技术，使CLIP模型训练样本减少470万。
原文摘要 · Abstract (English)
Large-scale web-crawled datasets contain noise, bias, and irrelevant information, necessitating data selection techniques. Existing methods depend on hand-crafted heuristics, downstream datasets, or require expensive influence-based computations -- all of which limit scalability and introduce unwanted data dependencies. To address this, we introduce the Mimic Score, a simple and geometry-based data-quality metric that evaluates utility by measuring alignment between a sample's gradients and a target direction induced by a pre-trained reference model. This leverages readily available model weights, avoids needing validation datasets, and incurs minimal computational overheads. Building on this metric, we propose Grad-Mimic, a two-stage framework that re-weights samples online to accelerate training and aggregates sample utilities offline to construct effective data filters. Empirically, we show that using mimic scores to guide training improves data efficiency, accelerates convergence, yields consistent performance gains across six image datasets, and enhances CLIP models with 20.7% fewer training steps. Additionally, mimic score-based filters augment existing filtering techniques, enabling improved CLIP models trained with 4.7 million fewer samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。