研究数据中的虚假关联如何影响小任务蒸馏的鲁棒性
Investigating the Robustness of Subtask Distillation under Spurious Correlation
- 在存在虚假关联的数据上测试蒸馏方法，考察其性能变化
- 先进方法如SubDistill在强关联下仍保持稳定，基线方法性能急剧下降
- 适合关注模型鲁棒性和真实数据挑战的研究者
子任务蒸馏是一种新兴范式，从大型通用基础模型中提取小型专用模型，以部署于资源受限或独立计算系统。尽管蒸馏依赖教师模型，但其训练数据往往规模有限、代表性不足或存在虚假相关性。本文评估了现有蒸馏方法及最新SubDistill方法在含虚假相关性的数据上的表现。随着相关性强度增加，SubDistill等先进方法保持相对稳健，而部分基线方法性能退化至接近随机水平。研究凸显了在非理想真实数据上应用知识蒸馏所面临的挑战。
原文摘要 · Abstract (English)
Subtask distillation is an emerging paradigm in which compact, specialized models are extracted from large, general-purpose 'foundation models' for deployment in environments with limited resources or in standalone computer systems. Although distillation uses a teacher model, it still relies on a dataset that is often limited in size and may lack representativeness or exhibit spurious correlations. In this paper, we evaluate established distillation methods, as well as the recent SubDistill method, when using data with spurious correlations for distillation. As the strength of the correlations increases, we observe a widening gap between advanced methods, such as SubDistill, which remain fairly robust, and some baseline methods, which degrade to near-random performance. Overall, our study underscores the challenges of knowledge distillation when applied to imperfect, real-world datasets, particularly those with spurious correlations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。