用类别中心点选样本，简单有效提升小样本知识蒸馏性能
Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
- 以每类样本平均图像为中心，选距离最近的样本作为核心集
- 在4个数据集上均超越随机选择和其他核心集方法，提升显著
- 方法极简可复现，适合做基线对比或快速部署
核心集选择旨在从大规模数据集中选出少量代表性样本以实现高效模型训练。该问题在小样本知识蒸馏（KD）场景下依然具有挑战性，此时一个全规模预训练教师模型指导学生网络。典型采样策略常难以超越随机选择基线。本文提出few-medoids，一种极其简单的核心集选择方法：选取每个类别中距离其类别中心（平均图像）最近的样本。我们在四个数据集上进行了广泛的KD实验，覆盖多种图像分类任务，并使用三对教师-学生模型（包含卷积和Transformer网络）。尽管方法极为简单，实验证明few-medoids始终优于随机选择及其他核心集策略。因此我们认为，few-medoids可作为未来核心集研究中常用基线（如herding或k-center Greedy）的即插即用替代方案。代码已公开于https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset。
原文摘要 · Abstract (English)
Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training. The problem remains challenging even in the few-shot knowledge distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。