通过动态选择高置信度图像，提升黑盒少样本知识蒸馏的多样性与准确率
Improving Diversity in Black-box Few-shot Knowledge Distillation
- 动态筛选教师模型生成的高置信度图像，实时融入对抗训练
- 在7个数据集上达到当前最优效果，显著提升学生模型精度
- 适合资源受限场景下少样本知识蒸馏，尤其适用于无内部访问的黑盒教师
知识蒸馏（KD）是一种有效压缩大型网络（教师）为小型网络（学生）的方法，性能损失极小。然而，多数KD方法依赖大规模训练数据和对教师网络的内部访问，这在实际中往往受限。为此，研究者提出更贴近实际的黑盒少样本知识蒸馏（black-box few-shot KD）设置：学生仅用少量图像和一个黑盒教师进行训练。现有方法通常生成合成图像，但缺乏主动策略以提升其多样性，而多样性是学生学习的关键。本文提出一种新型生成对抗网络训练方案，通过教师监督自适应选择高置信度图像，并实时引入对抗学习过程。该方法显著扩展并优化了蒸馏数据集的多样性，大幅提高学生模型准确率。在七个图像数据集上的广泛实验表明，本方法优于现有少样本KD方法，达到当前最佳水平。代码已开源：https://github.com/votrinhan88/divbfkd。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a well-known technique to effectively compress a large network (teacher) to a smaller network (student) with little sacrifice in performance. However, most KD methods require a large training set and internal access to the teacher, which are rarely available due to various restrictions. These challenges have originated a more practical setting known as black-box few-shot KD, where the student is trained with few images and a black-box teacher. Recent approaches typically generate additional synthetic images but lack an active strategy to promote their diversity, a crucial factor for student learning. To address these problems, we propose a novel training scheme for generative adversarial networks, where we adaptively select high-confidence images under the teacher's supervision and introduce them to the adversarial learning on-the-fly. Our approach helps expand and improve the diversity of the distillation set, significantly boosting student accuracy. Through extensive experiments, we achieve state-of-the-art results among other few-shot KD methods on seven image datasets. The code is available at https://github.com/votrinhan88/divbfkd.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。