用少量标注数据高效选出最适合的预训练模型。
All models are wrong, some are useful: Model Selection with Limited Labels
- 从无标签数据中选最有信息量的样本标注，指导模型选择。
- 在18组模型、16个数据集上，减少94.15%标注成本找最优模型。
- 适合资源有限时需快速选型的机器学习部署场景。
我们提出MODEL SELECTOR框架，实现标签高效的预训练分类器选择。给定一个无标签目标数据集，该框架通过采样少量高信息量样本进行标注，以高效识别出最适配该数据集的预训练模型。大量实验表明，相比最强基线，MODEL SELECTOR在18组模型、16个数据集（超1500个预训练模型）上，可将最佳模型选择的标注成本降低最多达94.15%；在选择性能仅比最优模型低1%的近优模型时，标注成本最高可降低72.41%，展现了其在模型选择中的鲁棒性。
原文摘要 · Abstract (English)
We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。