动态调整样本级知识蒸馏权重，提升长尾数据学习效果
Balancing Knowledge Distillation for Imbalance Learning with Bilevel Optimization
- 用双层优化框架为每个样本生成自适应的硬损失与软损失权重
- 在长尾CIFAR-10/100上，比现有方法在不同不平衡因子下均更优
- 适合处理标签不均衡场景下的模型压缩与知识迁移任务
知识蒸馏通过硬损失与软损失的混合，将高容量教师模型的知识传递给轻量学生模型。在数据不平衡场景下，固定比例的硬软损失组合会使学习过程变得脆弱。近期研究尝试在长尾设置中重新加权这两个组件，但多数方法未实现样本级自适应，也未考虑学生模型训练过程中的行为变化。为此，我们提出BiKD——一种双层优化框架，对每个样本动态平衡硬损失与软损失。采用权重生成网络，基于一个小规模均衡验证集生成自适应的样本级权重。学生模型在无约束的加权硬软损失组合下训练，允许其自主调节两个损失项。此外，我们提出多步SGD策略，更准确高效地优化权重模型。在长尾CIFAR-10/100上的实验表明，该方法在不同不平衡因子下均超越了近期平衡蒸馏方法。
原文摘要 · Abstract (English)
Knowledge distillation transfers knowledge from a high capacity teacher to a compact student using a mixture of hard and soft losses. On imbalanced data, a fixed weighting between hard and soft losses becomes brittle the learning process. Recent studies try to reweight these components in long-tailed settings. However, most of these methods do not adapt weights at the sample-wise level and do not take into account the students behavior during training. To address this, we propose BiKD -- a bilevel framework that dynamically balances hard and soft losses for each sample. We employ a weight generation network that produces adaptive per-sample weights, guided by a small balanced validation set. The student is now trained with an unconstrained combination of weighted hard and soft losses, allowing the student to relax both terms. We further propose a multi-step SGD strategy to optimize the weight model more accurately and efficiently. Experiments on long-tailed CIFAR-10/100 show that our approach surpasses recent balanced distillation methods across imbalance factors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。