arXiv:2609.05481cs.AI2026-09

提出一种更高效可靠的文本分类知识蒸馏方法,提升小样本下的模型性能。

RAPID: Reliability-Aware Pair Importance Distillation

  • 通过可靠性感知筛选重要关系对,动态优化蒸馏目标
  • 在AG News和SST-2上分别达到94.28%和88.80%准确率,优于基线
  • 适用于资源受限场景,尤其适合小批量文本任务的高效蒸馏

实例间关系蒸馏通过匹配小批量内样本间的关系来传递教师模型的表征几何结构。计算所有关系对具有与批次大小平方相关的复杂度,而均匀采样可能低效利用有限的关系预算。本文提出可靠性感知关系重要性蒸馏(RAPID),将可靠性门控的关系目标与全支持自适应关系提议分离。可靠性决定强调哪些教师关系,校准的教师熵与脱离的学生-教师残差决定评估哪些关系。精确的逆提议校正使损失和梯度估计器在门控小批量目标下条件无偏。我们在两个文本分类设置中评估RAPID:使用三组配对种子和256个关系预算的BERT到DistilBERT蒸馏(AG News),以及使用三组配对种子和64个关系预算的DistilBERT到DistilBERT蒸馏(SST-2)。可靠性门控关系蒸馏在两个数据集上均取得最高平均学生准确率:94.285±0.054%(AG News)和88.800±0.532%(SST-2)。RAPID排名第二,分别为94.241±0.025%和88.685±0.462%,显著优于交叉熵基线(94.154±0.124% 和 87.271±0.162%)。初步评估计入相同总预算。两个设置下,门控目标均带来最高均值准确率,而自适应提议保持在种子级波动范围内。结果支持将目标可靠性与评估优先级视为可分离的设计维度。

原文摘要 · Abstract (English)

Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently. We introduce Reliability Aware Pair Importance Distillation, or RAPID, which separates a reliability gated relational target from a full support adaptive pair proposal. Reliability determines which teacher relations are emphasized, while calibrated teacher entropy and detached student-teacher residuals determine which relations are evaluated. Exact inverse proposal correction makes the loss and gradient estimators conditionally unbiased with respect to the gated mini batch target. We evaluate RAPID in two text classification settings: AG News with BERT-to-DistilBERT distillation using three paired seeds and a relation budget of 256, and SST-2 with DistilBERT to DistilBERT distillation using three paired seeds and a relation budget of 64. Reliability gated relational distillation achieves the highest observed mean student accuracy on both datasets: 94.285 plus or minus 0.054 percent on AG News and 88.800 plus or minus 0.532 percent on SST-2. RAPID ranks second, achieving 94.241 plus or minus 0.025 percent and 88.685 plus or minus 0.462 percent, respectively, compared with 94.154 plus or minus 0.124 percent and 87.271 plus or minus 0.162 percent for the cross entropy baseline. Pilot evaluations are counted toward the same total budget as the main relation evaluations. Across both settings, the gated target yields the highest mean accuracy, while the adaptive proposal remains within seed-level variation. These results support the modular view that target reliability and evaluation priority are separable design dimensions.

知识蒸馏关系学习小样本文本分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。