针对视觉模型微调,提出按类别难易度不均匀选数据子集的新方法。
Non-Uniform Class-Wise Coreset Selection for Vision Model Fine-tuning
- 按类别难度动态分配选样预算,兼顾类间差异与样本质量。
- 在10个数据集上优于现有方法,精度更高且计算更高效。
- 适合需要高效微调大模型的研究者和工程师使用。
核心数据集选择旨在识别少量但信息量高的数据子集,从而实现更高效的模型训练并降低存储开销。近期,该能力被用于应对大型基础模型微调的挑战,为其实现高效实用部署提供了直接路径。然而,大多数现有方法是类别无关的,忽略了类别间的显著难度差异,导致过度删减过于简单或困难的样本,造成数据预算分配不当,最终降低核心数据集性能。为此,我们提出非均匀类别级核心数据集选择(NUCS),一种同时融合类别级与样本级难度的新框架。我们提出一个稳健的全局类别难度度量,以截尾平均值量化每样本难度得分。基于此度量,我们的方法在类别间进行理论依据充分的非均匀数据选择预算分配,并在类别内自适应选择最优难度范围的样本。在多种视觉分类任务上的广泛实验表明,NUCS在10个不同数据集和预训练模型上持续优于现有最先进方法,既实现更高精度又具备更强计算效率,凸显了非均匀类别级选择策略在推进大模型高效微调方面的潜力。
原文摘要 · Abstract (English)
Coreset selection aims to identify a small yet highly informative subset of data, thereby enabling more efficient model training while reducing storage overhead. Recently, this capability has been leveraged to tackle the challenges of fine-tuning large foundation models, offering a direct pathway to their efficient and practical deployment. However, most existing methods are class-agnostic, causing them to overlook significant difficulty variations among classes. This leads them to disproportionately prune samples from either overly easy or hard classes, resulting in a suboptimal allocation of the data budget that ultimately degrades the final coreset performance. To address this limitation, we propose Non-Uniform Class-Wise Coreset Selection (NUCS), a novel framework that both integrates class-level and sample-level difficulty. We propose a robust metric for global class difficulty, quantified as the winsorized average of per-sample difficulty scores. Guided by this metric, our method performs a theoretically-grounded, non-uniform allocation of data selection budgets inter-class, while adaptively selecting samples intra-class with optimal difficulty ranges. Extensive experiments on a wide range of visual classification tasks demonstrate that NUCS consistently outperforms state-of-the-art methods across 10 diverse datasets and pre-trained models, achieving both superior accuracy and computational efficiency, highlighting the promise of non-uniform class-wise selection strategy for advancing the efficient fine-tuning of large foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。