通过鞍点区域逼近,实现更高效的神经网络知识蒸馏。
SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

- 将教师模型视为优化曲率与领域代理,定位低损失鞍点区。
- 在疟疾血涂片分类中达94.8%准确率,比传统方法高24.7个百分点。
- 适合资源受限场景,如医疗设备部署,且收敛更稳定鲁棒。
现代深度神经网络虽具强大影响力,但参数量过大难以部署于医院设备、能源基础设施等低算力环境。主流知识蒸馏方法依赖模仿:小模型复制教师输出逻辑值,但实际表现差、收敛困难,仅起正则化作用而非真正知识迁移。本文提出鞍点招募蒸馏(SPRKD),将蒸馏从模仿重构为利用教师作为优化曲率和领域代理,通过嵌入与盆地分形特性识别具有强下降潜力的鞍点区域。基于海森矩阵特征值谱密度(ESD),SPRKD定位低损失鞍点区;弱教师集成生成近似鞍点区域(ASR),经注入式迁移学习重参数化至学生模型,并采用指数衰减欧氏变换、负海森特征步进和高斯扰动逼近。在6,430参数的CNN从25,546参数弱教师蒸馏的疟疾血涂片分类任务中,SPRKD达到94.8%验证准确率,较响应蒸馏高出24.70个百分点(McNemar p=6.3e-87),与同架构原生训练基线统计等价(p=1.0)。在MNIST、CIFAR-100和TinyImageNet上,性能超越原生训练基线最高达8个百分点。海森谱密度与二维损失景观分析表明,模型收敛至更宽极小值,海森迹与谱半径显著低于响应蒸馏与对照学生,体现更平滑下降路径与更强噪声鲁棒性。
原文摘要 · Abstract (English)
Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure. Predominant knowledge distillation (KD) methods favor replication: smaller students mimic teacher output logits, yet empirically yield low task performance, hamper convergence, and act merely as regularization rather than substantive knowledge transfer. We propose Saddle Point Recruitment for Knowledge Distillation (SPRKD), reframing distillation from replication to employing teachers as optimization-curvature and domain proxies, characterizing saddle points as regions of strong further-descent potential via embedding and basin-fractal properties. Using Hessian eigenvalue spectral density (ESD), SPRKD identifies low-loss saddle regions for student re-exploration; weak-teacher ensembles are aggregated into an Approximated Saddle Region (ASR), re-parameterized into the student via Transfer Learning by Injection, and approached with exponentially decaying Euclidean transformations, Negative Hessian Eigensteps, and Gaussian perturbations. On malaria blood smear classification with a 6,430-parameter CNN distilled from a weak 25,546-parameter teacher, SPRKD reaches 94.8% validation accuracy, outperforming Response KD by 24.70 percentage points (McNemar p = 6.3e-87) and matching scratch-trained baselines of the same architecture to statistical equivalence (p = 1.0). Across MNIST, CIFAR-100, and TinyImageNet, SPRKD exceeds scratch-trained baselines by up to 8 percentage points on preliminary benchmarks. Hessian ESD and 2-D loss landscape analysis show convergence to wider minima with substantially smaller Hessian trace and spectral radius than Response KD and control students, indicating smoother descent and greater noise robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。