学生模型容量影响知识蒸馏效果，小模型受益有限。
Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10
- 对比四种教师-学生结构，发现学生容量是影响蒸馏效果的关键变量。
- 特征蒸馏优于逻辑蒸馏，学生输出分布更接近教师，但小模型无显著提升。
- 模型架构改进比蒸馏增益更大，修正网络结构可提效5.5至7.2个百分点。
我们系统研究了在CIFAR-10上基于ResNet的教师-学生配对中，模型容量关系对知识蒸馏（KD）效果的影响。在四个教师-学生组合（R50→R18、R34→R18、R50→R34、R101→R34）中，采用严格评估协议：超参数与检查点在保留验证集上选择，配置经五次随机种子重跑，测试集仅用于最终报告。除准确率外，还通过教师-学生一致性和KL散度直接衡量蒸馏保真度。主要发现：第一，学生容量模式仍存在但幅度减弱；仅当学生为R34且使用特征蒸馏时，增益显著（+0.19和+0.21 pp，p<0.05），而R18学生无显著提升；第二，特征蒸馏在所有组合中表现不低于或优于逻辑蒸馏，其输出分布更接近教师（T=1时的KL值更低），尽管未直接观察教师逻辑值；第三，教师-学生前1准确率一致率全组平坦，表明保真度与准确率提升解耦；第四，架构设计主导蒸馏效果：对32×32输入修正残差块结构可带来+5.5至+7.2个百分点的增益，超过最大蒸馏增益的25倍。此外，原版本中报告的梯度裁剪缺陷实无影响，更大的增益源于测试集选择偏差。代码与结果见github.com/umutonuryasar/kd-capacity-gap（v2.0标签）。
原文摘要 · Abstract (English)
We investigate how teacher-student capacity relationships modulate knowledge distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10. Across four teacher-student pairs (R50->R18, R34->R18, R50->R34, and R101->R34) we compare Logit-KD and Feature-KD under a strict evaluation protocol: hyperparameters and checkpoints are selected on a held-out validation split, selected configurations are re-run with five seeds, and the test set is used exclusively for final reporting. Beyond accuracy, we measure distillation fidelity directly via teacher-student agreement and KL divergence. We report four findings. First, the student-capacity pattern survives the corrected protocol at reduced magnitude: the only statistically significant gains occur for R34 students under Feature-KD (+0.19 and +0.21 pp, p<0.05), in two pairs whose teachers differ two-fold in parameters but not in accuracy, localizing the moderating variable on the student side, while no KD gain for R18 students is distinguishable from zero. Second, Feature-KD matches or outperforms Logit-KD in all four pairs, and its students land closer to the teacher's output distribution (KL at T=1) than Logit-KD students despite never observing teacher logits. Third, top-1 teacher-student agreement is flat across all pairs, decoupling fidelity from accuracy gains. Fourth, architecture dominates KD: correcting the ResNet stem for 32x32 inputs is worth +5.5 to +7.2 pp, more than 25x the largest KD gain. We also retract an attribution made in v1: a controlled re-run shows the reported gradient-clipping bug had no measurable effect, and v1's larger gains are explained by test-set selection. Code and results: github.com/umutonuryasar/kd-capacity-gap (tag v2.0).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。