提出CoCaRS方法，提升异构知识蒸馏中特征冗余抑制效果。
CoCaRS: Correlation Calibration-Based Redundancy Suppression for Heterogeneous Knowledge Distillation
- 通过混淆证据估计与强度分配控制校准特征去相关性。
- 在CIFAR-100和ImageNet-1K上显著提升蒸馏性能。
- 自适应调节系数，降低对超参数设置的敏感性。
知识蒸馏（KD）使小型学生模型从强大教师模型学习，成为模型压缩的有效范式。多样化模型架构推动了从同构到异构知识蒸馏的发展。然而，教师与学生模型间架构归纳偏置的差异常导致显著的表示差异，限制直接知识迁移的效果。近期，冗余抑制为异构KD提供了新视角，通过去相关化师生特征关联来保持跨架构不变性并减少特征冗余。然而，该方法可能因统一去相关而削弱有用结构信息，且固定系数使冗余抑制的有效性对教师-学生组合和训练阶段敏感。为此，提出基于相关性校准的冗余抑制方法（CoCaRS），以更好保留结构信息同时抑制冗余，并降低对系数设置的敏感性。具体而言，CoCaRS通过混淆证据估计（CEE）和强度分配控制（SAC）校准特征去相关，分别捕捉可靠语义关系用于相关性估计，并在去相关过程中保留判别性结构。自适应系数调节（ACR）进一步根据相对损失尺度调节校准后冗余抑制目标的贡献，降低对系数设置的敏感性。在CIFAR-100和ImageNet-1K上的大量实验验证了CoCaRS在提升蒸馏性能和降低系数敏感性方面的有效性。代码将很快发布。
原文摘要 · Abstract (English)
Knowledge distillation (KD) enables a compact student model to learn from a powerful teacher and has become an effective paradigm for model compression. The emergence of diverse model architectures has extended KD from homogeneous to heterogeneous settings. However, differences in architectural inductive biases between the teacher and student models often result in substantial representation discrepancies, limiting the effectiveness of direct knowledge transfer. Recently, redundancy suppression has offered a new perspective on heterogeneous KD by preserving cross-architecture invariance and reducing feature redundancy through decorrelation of teacher-student feature correlations. Nevertheless, this formulation may weaken useful structural information through uniform decorrelation, while a fixed coefficient may make the effective contribution of redundancy suppression sensitive to teacher-student pairs and training stages. To address these problems, Correlation Calibration-based Redundancy Suppression (CoCaRS) is proposed to better retain structural information while suppressing redundancy and reduce sensitivity to coefficient settings across teacher-student pairs and training stages. Specifically, CoCaRS calibrates feature decorrelation through Confusion Evidence Estimation (CEE) and Strength Allocation Control (SAC), which respectively capture reliable semantic relations for correlation estimation and preserve discriminative structure during decorrelation. Adaptive Coefficient Regulation (ACR) further regulates the contribution of the calibrated redundancy suppression objective according to its relative loss scale, reducing sensitivity to coefficient settings. Extensive experiments on CIFAR-100 and ImageNet-1K validate the effectiveness of CoCaRS in improving distillation performance and reducing sensitivity to coefficient settings. Code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。