用多模态信息选数据,让模型训练更快更准。
A CLIP-Powered Framework for Robust and Generalizable Data Selection
- 利用CLIP融合图文信息评估样本重要性
- 在多个数据集上比现有方法性能更高
- 能自动剔除噪声数据,适合高质量数据筛选
大规模数据集推动了深度学习的进展,但其训练带来巨大存储与计算开销。现实数据常含冗余和噪声,影响训练效率与模型表现。数据选择可识别最具代表性样本,降低训练成本并缩小性能差距。现有方法多依赖单模态信息评分,对噪声或损坏样本评估不准。为此,本文提出一种基于CLIP的新型数据选择框架,通过多模态知识全面评估样本影响力,并结合三模块(数据集适配、样本评分、选择优化)实现多目标优化。实验表明,该方法在多个基准数据集上持续优于现有最优基线。尤其能有效剔除噪声或损坏样本,以更少数据实现更高性能,不仅加速训练,也提升整体数据质量。
原文摘要 · Abstract (English)
Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets invariably incurs substantial storage and computational overhead. Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance. Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs. Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples. To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. Specifically, our framework consists of three key modules-dataset adaptation, sample scoring, and selection optimization-that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization. Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。