探索知识蒸馏中替代数据集的可行性与标准
What Makes a Good Dataset for Knowledge Distillation?
- 测试多种替代数据源，包括合成图像
- 发现非真实图像也能有效传递知识
- 提出判断数据集优劣的关键标准
知识蒸馏（KD）是模型压缩的常用方法，通常假设训练学生模型时可访问教师模型的原始数据。但在持续学习或使用企业保密数据集训练的大模型场景下，原始数据可能无法获取。这促使研究者采用其他补充数据源，但效果参差不齐。本文探讨多种潜在的替代蒸馏数据集，证明即使非自然的合成图像也可作为有效的替代方案。通过分析这些数据，我们归纳出若干评估数据集适配度的标准，为实际应用提供指导。代码已开源：https://github.com/osu-cvl/good-kd-dataset。
原文摘要 · Abstract (English)
Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the teacher's original dataset will also be available when training the student. However, in situations such as continual learning and distilling large models trained on company-withheld datasets, having access to the original data may not always be possible. This leads practitioners towards utilizing other sources of supplemental data, which could yield mixed results. One must then ask: "what makes a good dataset for transferring knowledge from teacher to student?" Many would assume that only real in-domain imagery is viable, but is that the only option? In this work, we explore multiple possible surrogate distillation datasets and demonstrate that many different datasets, even unnatural synthetic imagery, can serve as a suitable alternative in KD. From examining these alternative datasets, we identify and present various criteria describing what makes a good dataset for distillation. Source code is available at https://github.com/osu-cvl/good-kd-dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。