arXiv:2606.12171cs.CVcs.LG2026-06

mixup与知识蒸馏结合,让模型更准且更可信。

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

  • 用mixup增强学生模型在邻近区域的线性结构
  • 准确率提升,置信度降低一个数量级
  • 适合需要可靠预测的场景,如医疗、自动驾驶

知识蒸馏(KD)和mixup均能提升类别边界的平滑性:前者捕捉概率分布中的类间关系,后者通过输入的凸组合强制该关系。然而,当mixup仅用于学生训练时,其交互机制仍不明确——此时教师被查询于训练中从未见过的邻近分布输入,这种可控的分布错配对知识迁移的影响尚未被阐明。我们发现,这种错配导致教师的监督信号被分布混淆主导,而非类间结构。尽管如此,学生并未简单模仿教师:它独立获得了更强的邻近区域线性,而教师本身不具备这一特性,实现了超越‘暗知识’的迁移。在CIFAR和ImageNet上,使用mixup的KD始终提升学生准确率,并使过自信程度降低一个数量级,且不同容量教师均有效。关键的是,校准能力可独立于准确率从教师传递给学生;温度缩放控制着可量化的准确率-校准权衡,且在邻近训练下更加显著。这些结果表明,mixup蒸馏并非标准KD的退化版本,而是同时塑造判别性能、不确定性估计与表征几何的更丰富迁移通道。

原文摘要 · Abstract (English)

Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.

知识蒸馏mixup模型校准可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。