提出新方法提升知识蒸馏中少数群体的性能
Adaptive Group Robust Ensemble Knowledge Distillation
- 用偏差模型筛选教师，选择对少数群体有益的知识
- 在多个数据集上显著提升最差子群体准确率
- 适合关注公平性与鲁棒性的模型压缩研究者
神经网络易学习数据中的虚假关联，导致少数子群体性能下降。现有研究表明，将知识从复杂教师模型蒸馏到较简单的学生模型时，这种差异会被放大。虽然集成学习可提升最差子群体表现，但当教师为去偏集成时，传统集成蒸馏反而会降低少数群体性能。本文提出自适应组鲁棒集成知识蒸馏（AGRE-KD），通过引入一个额外的偏差模型，选择梯度方向偏离该模型的教师，以增强对学生模型在未知少数群体上的帮助。实验表明，该方法在多个数据集上优于传统蒸馏与经典多数投票集成，且能有效提升最差子群体表现。
原文摘要 · Abstract (English)
Neural networks can learn spurious correlations in the data, often leading to performance degradation for underrepresented subgroups. Studies have demonstrated that the disparity is amplified when knowledge is distilled from a complex teacher model to a relatively ``simple'' student model. Prior work has shown that ensemble deep learning methods can improve the performance of the worst-case subgroups; however, it is unclear if this advantage carries over when distilling knowledge from an ensemble of teachers, especially when the teacher models are debiased. This study demonstrates that traditional ensemble knowledge distillation can significantly drop the performance of the worst-case subgroups in the distilled student model even when the teacher models are debiased. To overcome this, we propose Adaptive Group Robust Ensemble Knowledge Distillation (AGRE-KD), a simple ensembling strategy to ensure that the student model receives knowledge beneficial for unknown underrepresented subgroups. Leveraging an additional biased model, our method selectively chooses teachers whose knowledge would better improve the worst-performing subgroups by upweighting the teachers with gradient directions deviating from the biased model. Our experiments on several datasets demonstrate the superiority of the proposed ensemble distillation technique and show that it can even outperform classic model ensembles based on majority voting. Our source code is available at https://github.com/patrikken/AGRE-KD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。