用大模型提取概念瓶颈,高效选出能保持性能的极小训练子集。
Coreset Selection via LLM-based Concept Bottlenecks
- 基于大模型生成的概念瓶颈计算样本难度,无需训练下游模型。
- 在CIFAR和ImageNet上,所选子集性能优于随机采样和传统方法。
- 支持无标签数据,适合快速构建高性价比训练数据集。
核心集选择(Coreset Selection, CS)旨在从训练集中筛选出一个子集,使模型性能接近使用全部数据时的表现。现有先进方法依赖于训练动态得分,需先在全量数据上训练下游模型并记录其行为变化,计算成本高且难以解释,无法区分样本是普遍难学还是仅对特定模型难学。本文提出一种新方法:利用大语言模型提取的、人类可理解的文本属性(概念),独立于下游模型评估样本难度。具体通过训练线性概念瓶颈层,衡量样本视觉特征与概念瓶颈的对齐程度,进而计算难度得分,并基于该得分进行分层采样构建核心集。关键优势在于无需任何下游模型的完整训练即可高效计算得分,适用于无标签数据,且在多种下游模型上表现优异。在CIFAR-10/100和ImageNet-1K上的实验表明,本方法生成的核心集在高剪枝率下仍显著优于随机子集,性能可媲美甚至超越基于训练动态的方法。
原文摘要 · Abstract (English)
Coreset Selection (CS) aims to identify a subset of the training dataset that achieves model performance comparable to using the entire dataset. Many state-of-the-art CS methods select coresets using scores whose computation requires training the downstream model on the entire dataset first and recording changes in the model's behavior on samples as it trains (training dynamics). These scores are inefficient to compute and hard to interpret, as they do not indicate whether a sample is difficult to learn in general or only for a specific downstream model. Our work addresses these challenges by proposing a score that computes a sample's difficulty using human-understandable textual attributes (concepts) independent of any downstream model. Specifically, we measure the alignment between a sample's visual features and concept bottlenecks, derived via large language models, by training a linear concept bottleneck layer and computing the sample's difficulty score using it.We then use stratified sampling based on this score to generate a coreset of the dataset.Crucially, our score is efficiently computable without training the downstream model on the full dataset even once, leads to high-performing coresets for various downstream models, and is computable even for an unlabeled dataset. Through experiments on CIFAR-10/100, and ImageNet-1K, we show that our coresets outperform random subsets, even at high pruning rates, and achieve model performance comparable to or better than coresets found by training dynamics-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。